Network & Academic Computing Services
School Computing Coordinators
May 8, 2002

Agenda:

• NACS Virus scanning on the MTAs
  Brian Roode, Keith Chong
• Network security strategies, e.g. port blocking
  Garrett Hildebrand, Mike Iglesias

Summary:

Lively discussion centered on network security, with focus on the agenda items. Group resolutions include:

• SCCs support NACS' plan to throttle down bandwidth consumption by unknown high bandwidth consumers.
• SCCs support NetBIOS port blocking as a campus network security measure. The group requests advance warning in order that proper planning can be done.
• SCCs agree to registration of designated high-bandwidth servers.
• SCCs request NACS assistance in determining VPN solutions.

Attendees:

Discussion:

Prior to calling the meeting to order, conversation centered on the issue of blocking NetBIOS ports at the campus border router. Most in the room were in favor of the security measure, but worried about unseen negative impact in the userbase. Jason Valdry said it might cause problems. Kevin Ansel and Ted Roberge noted that graduate students routinely join campus domains and mount shares from their homes in graduate housing, for academic purposes; nearly all of Housing’s misuse comes from the undergraduate housing.

Some expressed a desire to proceed with school- or department-based blocking if campus blocking wasn’t feasible.

NACS Virus scanning on the MTAs
Keith Chong, Brian Roode

On Monday May 3, a new MTA (message transport agent) machine was installed, mta1.service.uci.edu. This MTA scans all mail for known viruses before delivery. Full details on the virus scanning, including NACS criteria for scan/clean/delivery/bounce, please see:
http://www.nacs.uci.edu/email/virus-scanning.html

Until now the campus had five MTA machines, none of which scanned for viruses. As of Monday May 3, the first virus scanning machine was installed. Over the next several days the five machines will be replaced with three higher-powered SunFire 280R machines, all of which will include the virus scanning software. The virus scanning software is Sophos’ MailScanner with SAVI engine.

The anti-virus pattern files on the MTAs are updated once per day. We can update more frequently, and manually if necessary.

Robert Boyd – Is there a performance hit affecting the speed of mail delivery?
A: The addition of scanning and cleaning certainly adds time of processing, but we hope that the faster machines will negate any delay. Please report any out-of-ordinary mail delivery delay to nacs@uci.edu.

Jason Valdry – Should we block exe & bat extensions as well
Bill Cohen – EXEs are often legitimate, such as drivers and updates sent by vendors.
Greg Ackerman – At the College of Medicine we block all EXEs, ever since we started virus scanning.
John Clarke – We block them at GSM too. One faculty member challenged this policy, but the resulting judgement was in the policy’s favor.

Garrett Hildebrand – If there is a question of the UC Computing Use policy, questions can be referred to Garrett or Mike Iglesias, who in turn can pass it on to a UC-wide committee of network security personnel, which includes authors of the UC policy. The committee usually rules in favor of security.

Kevin Ansel – Blocking EXEs could become a problem during the holidays. That’s when we see a lot of screensavers, backgrounds and e-cards being sent around.

Andrew Laurence – Have any users received a message which had been cleaned and misinterpreted the resulting “this message has been cleaned” text attachment to mean that their computer now has a virus?
No such instances reported.

Question – Will NACS block files with the .vbs extention?
Scott Talkovic – Scott suggested disabling the Windows scripting host via a registry hack, in order to completely remove this vulnerability.

Greg Ackerman – What messages, if any are sent to the recipient and sender if a virus is found?
Keith Chong – If we find a virus, we attach a text or html attachment (depending on the format of the original message) which explains that a virus was found and removed. Also included is information about the virus in question.

Diane Bisom – Can we see samples of warning messages, in order to give the users advance warning of what they might see?
NACS will provide samples to the SCC list and post to the NACS web site.
Ted Roberge – Thanks.

Scott Talkovic - Can we use the Sophos SAVI/MailScanner on our own mail servers?
Bob Hudack – Yes.

NACS has purchased a license for the Sophos mail gateway products (SAVI/MailScanner) which covers every person at UCI. The software can be installed on any number of mail servers in any department, on any (supported) platform. It is effectively a site license for mail gateways. This license is underwritten by NACS for use in your department.

Because all users are already covered for email, the Sophos desktop products are available as a “desktop upgrade” for $2.68 per desktop.

Bob can burn and send copies of SAVI/MailScanner to departments who request the software.

At request of the group, NACS will schedule a configuration kitchen for departments who want to install SAVI/Mailscanner on their own mail servers.

Diane Bisom – With email virus removal, will we still need to license software for the desktops?
Group Discussion: – Desktop protection is still the wise course of action, given that viruses can arrive via disk, file servers, web download, etc.

Jason Valdry – Will we have spam filters?
Answer: Marking or filtering spam is a much trickier issue. Spammers have gotten very good at masking their trail. While the MTAs used to reject an enormous amount of mail because of unregistered DNS origination and the like, today very little mail gets bounced for those reasons. The danger of a spam filter system is in false positives, where legitimate mail is wrongly marked as spam.

Andrew Laurence – From the first-day sample provided by Brian Roode and Keith Chong, about 4.5% of all email delivered to uci.edu carries a virus.

(For the period of Monday 5/6/2002 through Tuesday 5/14/2002, 7.3% of messages carried a virus.)

Bob Hudack will send a summary of virus software in use by UCI departments to the SCC list.

Campus Network Security
Garrett Hildebrand, Mike Iglesias

On April 25 2002, Mike Iglesias sent a request for comments to the UCICSCG mailing list, on the possibility of blocking NetBIOS ports at the campus border router, as a security measure. In this discussion we want to examine campus network security in light of the traditional open educational network. “Are we undergoing a sea change?” Mike Iglesias begins with a summary of why port blocking is being discussed.

Mike Iglesias – A large number of Windows systems have been, and are continuing to be hacked from outside UCI’s network. The attacks most often automated attempts to login remotely as the Administrator; when they succeed they then install warez software or bots which enable the hacker to utilize the machine for various purposes. These attacks most often succeed because of weak or no passwords on the Windows system being attacked.

Dennis Wiedeman – I was contacted by John Lenning regarding a system in our area. Can you illuminate why?

Garrett Hildebrand –

We have two network issues under examination, the first is remote hacking and possible port blocking. Second is the monthly bandwidth fee that UCI pays for traffic to/from the commercial Internet. This charge has risen sharply in the last several months, from $8,000/month to a the most recent $16,000.

NACS Director Dana Roode has directed NACS’ Network Operations to determine where the bandwidth is going. For the purposes of this analysis, a “high bandwidth host” is defined as using a weekly average of 0.5 megabits per second (Mbps). Analysis has shown that between 30 and 40 machines utilize about 25% of the total campus usage. Under the belief that any “unexpected high bandwidth hosts” are probably dealing in ‘junk’ traffic (either via hacking or deliberate installation of P2P applications), Network Operations has been directed to contact the local computing personnel to investigate. Any “unexpected high bandwidth host” will be limited at the border router to 0.1Mbps; if such a host is determined to have legitimate needs for high bandwidth, the limit will be removed.

I suspect John Lenning’s inquiry to RGS was in this second category, regarding a high bandwidth host.

Dennis Wiedeman – What kind of uses are you seeing on the ones that aren’t servers?
Answer – Mostly warez, movies and MP3s. A lot of P2P applications such as Gnutella, KaZaA and Morpheus. Illicit FTP servers, etc.

Garret Hildebrand demonstrated the UCInet Metrics page, which lists displays snapshots of bandwidth utilization by department. “Internet” traffic is on the commercial Internet, for which UCI pays per-bandwidth usage. “Calren” traffic is traffic to other educational institutions on the CALREN and Abiline (Internet2) networks).

The group discussed the practicality of throttling all UCI hosts at .1Mbps, except for known “registered” servers. Garrett and Mike agreed that it was do-able in the practical sense, but would probably impede academic needs somewhere unforeseen.

Ted Roberge – In Residential Housing, we already block all outbound server traffic.

Garrett Hildebrand – In fact, we're not worried about the typical server. Most campus servers are well below the .5 Mbps mark. We’re not worried about short-term spikes in bandwidth, but long-term high utilization such as the P2P and warez machines. For example: andromeda.acs.uci.edu (aka ftp.uci.edu) , which serves a world-readable mirror of Red Hat Linux, used only 4.3 Mbps during the week of 5/6-5/12/2002.

Mike Iglesias – Registering servers might be administratively difficult. There are over 2000 web servers on campus.

Ted Roberge – We’ve had great success using a Packeteer packet shaper to isolate bandwidth flows in undergraduate housing. We slow the P2P stuff to just a trickle and make sure there’s plenty of bandwidth for web and email.
The worrisome traffic is only in undergrad housing. Graduate housing is completely different, they’re too busy to bother with that stuff. Graduate traffic is burstey, like when they download large data sets.

Kevin Ansel – If any one is interested, we have an extra Packeteer we can loan. Just run it in diagnostic mode, and it’ll map out where your bandwidth is going. It’ll have to be coordinated with NACS, but we’re happy to help.

Scott Talkovic – Can we block ports only on incoming connections? So that incoming attempts are rejected, but it’s possible to originate a connection from on campus?
Mike Iglesias – Yes

Jason Valdry – How many PCs are affected by NetBIOS attacks? I’m concerned that we’re talking about dealing with only a small symptom of a much larger issue.
Answer – Thousands of access attempts occur daily.

John Clarke - Is now the time to start a campuswide VPN? We’re all clearly spending a great deal of support time and money on these issues, when a broad security measure like port blocking or VPN could solve the problem. I asked on of our alums who works at Microsoft what he thought about this, expecting him to say it’d be the worst possible thing, but he was surprised we hadn’t done it already. What about internal attacks?

Garrett Hildebrand – Blocking NetBIOS ports at the border really only addresses a known point of high vulnerability. Security isn’t just one thing, and unfortunately we’re not a corporation with a closed network and firewalls. We’re finding that UCI’s traffic outstrips the capabilities of the Cisco PIX firewalls, which are rated for 1 Million connections. We’ve seen 1.3 Million.

Andrew Laurence – We all know that blocking NetBIOS ports will cause some amount of disruption in all our organizations. Will it be catastrophically crippling for anyone?
All agreed that there would be some disruption. No one anticipated catastrophic results.

John Romine – Can NACS help me find these machines? Since anyone on campus can request an IP number and DNS name, machines get set up all the time without my knowledge. If a machine in Engineering winds up in the “high bandwidth” list, it’s not likely that I’ll be able to physically find it. Can we have an easy way to register servers?

John Clarke – Are we going to force registration of servers?
Answer – We’re not sure yet. Right now we’re just trying to clamp down on the bandwidth abusers.

Diane Bisom – Can you give an example of some high bandwidth machines?
Garrett Hildebrand – andromeda.acs.uci.ed u (aka ftp.uci.edu) is just below the arbitrary threshold of 0.5 Mbps. Neither antpac.lib nor www.lib are over that threshold.

Greg Ackerman – The Windows command line utility “nbtstat –a ” can be very userful for locateing & diagnosing problems on computers with NetBios enabled. (Use nbtstat /? At Windows command prompt for help)

John Clarke – Well, our primary mission is to facilitate academic use and academic purposes. If this university resource is being jeopardized by misuse, we should act to protect the academic mission.
Greg Ackerman – I agree.

Ted Roberge – We’re looking into Cisco’s web caching equipment in order to reduce the amount of web traffic. But analysis will be needed before this can be done properly to save money.

Garrett Hildebrand – KaZaA is a very large consumer of bandwidth, and it’s nearly all illicit material. Dana Roode has directed Network Operations to limit all KaZaA traffic, campus wide, to 10 Mbps, and no more than 65 kbps per machine. This alone has reduced the campus bandwidth a great deal.

Ted Roberge – KaZaA traffic was overwhelming our Packeteer, we had to completely block its default port, port 1214. Once we put in these measures though, students figured out that they could take their laptops to UCInet Mobile Access at CornerStone or the Library and bypass Housing’s Packeteer packet shaping.
Mike Iglesias –This traffic is now now blocked on UCInet Mobile Access.

Andrew Laurence – Does the group support the blocking of NetBIOS ports as a security measure?
SCCs – Yes, so long as we have proper notice. It would be helpful to us if the argument is put forward in budget terms of opportunity cost for campus support efforts, staff and equipment.

Andrew Laurence – The group seems to be in favor of registering servers, and throttling of bandwidth abusers. Am I correct in these observations?
SCCs – Yes.

Diane Bisom – How should we register our servers with NACS?
Andrew Laurence – At this early stage, NACS will provide something that resembles a process. If nothing else, you’ll be able to tell us what machines to not throttle, if they show up on the high bandwidth list.

Bill Cohen – We should start an initiative for serving our customers, once port blocking is in place. What alternative methods for accessing files can we recommend? Are campus or departmental VPNs the solution?

Scott Talkovic – Is internet radio a problem?
Answer – Not yet, although broadcast.yahoo.com shows up a lot. (Streaming of professional sports.)

– Meeting Adjourned –

Network & Academic Computing Services

Updated: May 23, 2002

University of California, Irvine