CCC/CERN/96/05
HEP - CCC
PRESENT: M. Aguilar-Benitez, C. Berger, R. Devenish, L. Foà, H. Frese (for Gensch), I. Gaines, J. Ganouna, W. Hoogland, D. Jacobs (secretary), G. Kalmus (chairman), M. Kasemann, J.P. Repellin, F. Ruggieri, M. Turala, E. Valente, P. Villemoes, D.O. Williams
EXCUSED: E. Fernandez, U. Gensch, G. Goggi, L. Price, H. Wenninger
INVITED: H. Albrecht (for item 10), M. Delfino (for item 4), J. May, J. Shiers (for item 8)
The Chairman noted that U. Gensch was being replaced by H. Frese at this meeting and that apologies had been received from E. Fernandez. He welcomed Juergen May to the meeting, this time as a guest but soon to become a HEP-CCC member when he assumes the leadership of CERN's new computing division (replacing CN) in January 1997.
The agenda (CCC/CERN 96/04, version 5) was adopted without modification.
The Minutes were approved without comment.
The Chairman reviewed the outstanding actions from the previous meeting:
35. 2/96. HEP-CCC recommendations on Email handling. Gaines reported that Price had brought the views of HEP-CCC to the ESnet Email committee, which has now adopted the same standards as recommended by the HEP-CCC. Status: closed.
37. 2/96. Update on status of the TEN-34 project. On the agenda of the present meeting (item 5). Status: ongoing, kept on future agenda list.
Action : P. Villemoes, Secretary
33. 2/96. Create a HEP-CCC Web page. Jacobs has created a HEP-CCC home page as requested. The URL is http://nicewww.cern.ch/~Djacobs/Hepcccw3/HEPhome.htm
It is not linked to by the main CERN home page, nor does it yet contain a link to the HTASC pages.
There was a short discussion on whether the HEP-CCC page should be treated as public or private. It was decided that, for the time being at least, it should be kept simple (i.e. no password protection) but also private in the sense of not being linked to from the main CERN page. Status: closed.
38. 2/96. Use of groupware tools by RD44. RD44 will be asked to make a presentation on its use of groupware tools for software development at a future meeting. It was proposed to schedule this item at the 18/4/97 meeting. Status: pending, retained on the future agenda list.
Action : Secretary
39. 2/96. Tools for distributed editing of papers. Kasemann reported that HTASC has made a start on this but a report will not be available before the 18/4/97 meeting. Status: ongoing.
Action : HTASC
40. 6/96. Add HTASC members to the HEP-CCC mailing list. This has not been done. It was decided now that Kasemann will instead forward HEP-CCC mail to the HTASC members. Status: closed.
41. 6/96. Gather information on 4th Framework project applications. There had been no progress with this item. Devenish noted that there is some urgency since the closing date for the next round of proposals is February '97. He added that information on successful proposals is available on a Brussels Web page (URL since distributed to members and linked to from the HEP-CCC home page - Sec). HEP-CCC still holds the opinion that harmonisation of HEP proposals is valuable and decided to ask P.G. Innocenti to address the 18/4/97 meeting on this topic, if possible providing written information earlier. Status: ongoing, transferred to Secretary.
Action : Secretary
42. 6/96. Test of Transatlantic pilot. Frese said that he would report on this under item 7. Status: closed.
43. 6/96. Software licensing. On the agenda of the present meeting (item 8). Status: closed.
44. 6/96. INFN plans for EDMS. Ruggieri reported that there are no defined plans as yet. He undertook to inform HEP-CCC when there is some progress. Status: ongoing.
Action : Ruggieri
45. 6/96. CEDAR update. Invite C. Hauviller to present an update on the CEDAR project at the first HEP-CCC meeting in 1997. Status: ongoing, retained on future agenda list.
Action : Secretary
46. 6/96. FOCUS. On the agenda of the present meeting (item 4). Status: closed.
Commenting on Item 4 in the minutes (page 4, paragraph 7), Frese noted again that he would be able to give an update on the transatlantic ATM pilot between DESY and LBL under item 7.
Commenting on the same paragraph, Valente reported that, although there were now 15 projects approved by the G7 in the framework of GIBN, most of which seemed to be going well, that for HEP (proposed by himself) had made little concrete progress since the last HEP-CCC meeting. There appeared to be funding difficulties with the US partners.
A copy of Kasemann's overheads has been distributed to members.
Kasemann reported on the outcome of the 3-4 October meeting of HTASC at CERN. Apart from the usual reports, there had been a discussion of Windows NT (WNT) co-ordination (see below). The reflections on distributed collaborative work continued and he hoped for a report at the next meeting.
At an organisational level, the RECFA delegates of Finland, Portugal and Sweden have still to propose HTASC members. A list of names to form the network monitoring subcommittee has been assembled and plans are now moving ahead for a first meeting.
The situation for the European HEP DECnet Subcommittee (HDS) was as already reported to members in Email from D. Kelsey. Valente confirmed that native DECnet access to the USA, which has been running without guarantee via INFN since April '96 will most likely have to be turned off at the end of March '97 due to a move in Italy towards a B-WiN style of infrastructure. Speaking of the future of HDS, Kasemann said that it still has a role to play in co-ordinating documentation but that it probably needs to meet less frequently, if at all. Valente emphasised how well this organisation had functioned in the past and suggested that it could serve as a suitable model for handling similar needs in the future.
A discussion on PC usage had revealed a wide interest in many HEP laboratories for using both commodity hardware and software. Two operating systems were generating interest: Linux and WNT. For Linux advantages are that it integrates well with other UNIX environments and capitalises on the large body of UNIX experience in the community. There are, however, big doubts about its support status (freeware), stability and the availability of Fortran compilers. For WNT, the central management features are seen as good and the integration into the existing UNIX environment as reasonable (in the sense of using a WNT system as an X-terminal). AFS is not, however, currently available for WNT (although there is DFS support). Overall, the use of PCs in HEP is taking off rapidly. The provision of central management and services for this platform is seen as essential, along with central support of hardware and software.
At DESY, two large experiments are currently equipping themselves with PCs: HERMES, which will use Linux, having translated its simulation code with an f2c converter; ZEUS, which is moving its reconstruction code to WNT (using the Watcom Fortran compiler). The HERMES use of Linux is recognised to be problematic from the support point of view but is felt to be an acceptable solution for well-specified programs. In answer to Gaines, who asked about the performance obtained, Kasemann said that ZEUS found its Pentium Pro machines to be about twice as fast as its SGI Challenges (at less than a quarter of the cost). DESY acquires its PC's at the rate of about one per day from a small group of local vendors who supply the current "best buy" brands.
HTASC had also addressed the need for HEP-wide co-ordination of operating systems for commodity hardware. For UNIX, where the main contender is Linux (along with some FreeBSD and SCO_Unix use), the recommendation made is that any co-ordination that is necessary should pass via HEPiX. From the Windows family of systems, WNT is clearly felt to be the one that is interesting. Experience with WNT is rapidly building up in several labs. There is an obvious value in sharing expertise on (e.g.) setting up WNT domains within the HEP community but HTASC feels that any HEP-NT organisation can be kept very light-weight and should concentrate on coordination tasks. Kasemann cited the WNT Web pages at DESY, Oxford Univ. Physics Dept. and CERN. D. Kelsey has already set up a UK co-ordination group for WNT within the framework of the UK HEP Computing and Networking Advisory Panel (CNAP). HTASC feels that there is a clear need for a HEP-NT group to be set up now in order to collect experience at different sites and to give recommendations on a number of points. HTASC has drawn up a proposed HEP-NT mandate listing the points that should be addressed. Kasemann concluded by showing a list of things that HTASC thought HEP-NT should specifically not do. These include undertaking any development or porting tasks, ongoing software evaluations, or attempted restrictions on the software that users can run.
In answer to Berger, who expressed reservations about becoming dependent on Microsoft as operating system supplier, Kasemann said the UNIX would continue to be used and would be looked after by HEPiX. At the same time, Devenish pointed out that Linux is not really free, requiring considerable human resources in order to be usable in the HEP environment. Kalmus steered the discussion away from this aspect, saying that it is not HEP-CCC's role to pass judgement on one system or the other but rather to facilitate the choices that the community is making by itself.
Gaines said that there is a lot of WNT interest in the US HEP community, mentioning Florida State and also FNAL, where D0 is converting to WNT its trigger software (from VMS) and also its off-line code, while a Computing Division project is qualifying systems for future farm acquisitions. There are still questions about the true cost-effectiveness of WNT systems, with a feeling that the prices are comparable to UNIX when like is compared with like. Since there is a deliberate policy of avoiding monopoly situations, there is no chance that WNT will be adopted as the unique standard. There is no large US co-ordination effort for WNT. Away from the physics data processing field, WNT is seen as working well in servers supporting desktop machines.
Williams thanked Kasemann for the clear presentation and said he was in agreement with nearly all that had been said. He commented that it would be useful for meetings of HEP-NT and HEPiX to overlap in order to help the flow of information in both directions. He also felt that one should beware of polarising the community into UNIX and WNT camps and warned that, while initial experience with WNT as a physics computing platform had been good, the scaling to larger environments was not yet entirely proven. Hoogland supported the close contacts between HEPNT and HEPiX. However, when Kalmus suggested that the mandate of HEPiX might simply be broadened to encompass WNT, Kasemann said that HTASC was against this proposal on the grounds that HEP-NT was felt to require a lighter weight approach. Others nonetheless repeated the warnings about polarising the community and some thought that the memberships of HEPiX and HEP-NT should overlap. Williams stressed the importance of choosing a good HEP-NT chairman who could collaborate closely with Alan Silverman, the chairman of HEPiX. In passing, he sought to correct the idea that HEPiX had developed lots of software. This kind of activity had been restricted to scripts. He was sure that there would be an ongoing need for the two committees to discuss together on many matters such as security and batch support.
Summing up, Kalmus noted that HEP-CCC is clearly in favour of establishing a HEP-NT group as a sub-panel of HTASC and encourages HTASC to proceed with setting it up, consulting with HEPCCC on the choice of chair.
Action: Proposals for HEP-NT Chairman
Kasemann is asked to come to the April '97 HEP-CCC meeting with names to propose as HEP-NT chairman. Status: new.
Action : Kasemann
HEP-CCC also encourages HEPiX to make an investigation of Linux.
It was decided not to attempt to similarly transform HEPiX into an HTASC sub-panel since its coverage is world-wide rather than purely European. It was pointed out that HEP-NT will also need good interactions outside Europe and Foà proposed that these be handled via the LHC collaborations. Kalmus added that HEP-NT should certainly disseminate information as widely as possible and should also be free to enlist members from outside Europe if appropriate.
A copy of Delfino's overheads has been distributed to members.
He began by noting that the committee had met for the first time in March '96, with a mandate from the CERN DG to look at the match between the needs of CERN experiments and the evolution of the CERN computing services with a 2 year sliding window into the future. He noted that at least one CERN Director was present at each meeting, an important point in view of the resource implications that the recommendations of FOCUS might have.
FOCUS seeks to be more general in coverage than its predecessor committees MEDDLE (dominated by considerations of LEP) and COLLECT (LHC). There are about 35 members, chosen from both the user community and the service providers. The former are present as representatives of their experiment or physics programme sector rather than ad personam. The committee attempts to work out clear definitions of requirements and services. These definitions will hopefully be made available on the Web in the future. The minutes are kept deliberately short.
Liaison with the LCB is ensured by Delfino being a member of the LCB and the LCB chairman being invited to FOCUS meetings.
FOCUS meets about four times per year in closed session, working to agendas that are, in principle, defined about one month in advance. An attempt is also made to circulate working papers well ahead of the meetings. The philosophy is to resolve contentious issues by successive cycles of feedback. By comparison with the predecessor committees, a relatively large amount of time is devoted to discussion. The option to hold open sessions has yet to be taken up but it is likely that one may be held in July '97. In the spectrum of committees discussing HEP computing matters, Delfino sees FOCUS sitting between HEP-CCC (international co-ordination) and LCB (longer term future) on the one hand, and COCOTIME (CERN resource allocation for the next year) and the Desktop Forum (CERN desktop support) on the other. He said that all FOCUS documents are available from the secretary, Monique Budel (CERN-ECP).
Delfino next reviewed the agendas of the four FOCUS meetings that have taken place.
At the first meeting, on 14 March, Alan Norton had given a review of central data recording, not just for experiments but also for LHC test beams. FOCUS had concluded that CDR is clearly economic and had recommended it as the preferred mode of working. The presentation of Les Robertson on tape automation had led to the establishment of the Tape Automation Review Board (TARB) as a FOCUS sub-group. The TARB report had subsequently been discussed and was expected to lead to the placing of an order in December (subsequently adjudicated to STK - Sec.). Delfino noted that the new robot would be operated as a closed shop with import/export facilities. It was recognised that DLTs are presently the preferred media for off-site data exchange. The meeting had also heard a presentation on ATLAS simulation and had set up a study group to investigate the needs for future VMS support, a matter which remains to be resolved.
The next meeting, on 3 June, was less successful and did not manage to reach any conclusion on WNT. More time was taken by presentations: on the CERNVM run-down, Workgroup server use by CMS, consolidation of the mail service and the transition to CVS for code management.
The third meeting, on 9 September, had raised no new issues but had returned to TARB and the VMS support question.
The most recent meeting, on 14 November, had discussed file systems, with the provisional conclusion that AFS will be around for some years to come but that resources should be found to set up a DFS cell in order to check the suitability of this system and to be ready to implement it if needed. As a result of a discussion on batch systems in which the shortcomings of NQS and LoadLeveler had been debated, it had been decided to launch a pilot project with at least one LHC experiment and one from LEP in order to test the capabilities of LSF.
At this point, Ruggieri asked if Condor had been considered. Delfino replied that the features of Condor were not seen as matching the requirements looked for. In addition, its public domain status meant that its support level was not felt to be adequate. Hoogland questioned the necessity to optimise batch systems given the decreasing cost of hardware but Delfino insisted that the capacity available to HEP is still budget limited. He emphasised that LSF is a commercial product and has not been developed with HEP resources.
Moving on, Delfino remarked that it had not been possible for FOCUS to be fully synchronised with COCOTIME this year. He hoped that this situation would improve in '97 and that FOCUS could produce sets of recommendations for '98 and '99 in July, in time for the '98 COCOTIME exercise. These would most likely form the agenda of the tentatively planned open meeting.
Concluding, Delfino emphasised that FOCUS should not seek to micro-manage users or providers. With this in mind, he saw some future issues that should be debated in strategic terms as being consolidation of UNIX services, modernisation of the physics desktop machines and emerging themes such as storage management, WNT and future processor farms.
Hoogland enquired about the pressure to move from AFS to DFS. Delfino said that he was not aware of any pressure from the users, whose needs were basically defined as being a "good" AFS service. AFS does, however, rely on a single, small vendor, while DFS has the advantage of multi-vendor support (although SGI and SUN support is missing at the present time). AFS client desktop machines can access DFS servers. Hoogland observed that the adoption of WNT would undoubtedly increase the pressure for DFS.
Kalmus enquired if HEP-CCC members could receive the FOCUS minutes. Delfino replied that these are public once approved and will, in principle, be made available on Web pages. Jacobs undertook to provide a link from the HEP-CCC home page to the FOCUS pages when these are set up (now implemented - Sec.).
HEP-CCC agreed that it should receive regular annual reports from FOCUS, normally for the summer meeting (June/July).
Action: Annual reports from FOCUS
HEP-CCC should receive regular annual reports from FOCUS, normally for the summer meeting (June/July). Status: new, added to future agenda list.
Action : Secretary
A copy of Villemoes' overheads has been distributed to members.
Villemoes reported that the European Commission had approved the description of the project (Milestone 1), an act that de-blocks 5 MECU of EC funding and is based on the report of a review panel. This concluded that TEN-34 should continue since the potential benefits outweigh the risks, that statistics on non-research traffic should be provided, that planning for the post-TEN-34 era must start immediately and that the project should restore its focus on advanced features. The short life foreseen for TEN-34 is in view of the very high likelihood that alternative suppliers will soon emerge.
Villemoes pointed out that the identification of non-research (=commercial) traffic requires the definition of an "acceptable use" policy, that the end-date for TEN-34 is 31/8/98 which is not far off and that by "advanced features" was meant "native ATM" (the project presently being for IP over ATM). The review panel appeared to have been unaware of the 5 MECU allocated to JAMES for native ATM studies.
The national research networks now have a consortium agreement ready - a major achievement. This also includes cost distribution and the definition of signature rights for contracts. In addition the cash flow problems resulting from late reimbursement by the EC have been solved.
Next, Villemoes reviewed the TEN-34 organisation and money flow. At the top level is the EC RE1009 agreement, providing 40% of the funding. Within this, the consortium agreement (to be signed within weeks) defines the money flow. The national research networks are authorised to make the contracts with suppliers. As an example, 22 Mbps access will cost 3.1 MECU/year (of which the EC will pay 40%). Pricing is linear in this speed region. The contracts comprise a Unisource part which is almost ready, and FUDI parts which are being produced (these include technical details and consider connections as individual half-circuits). On a map of Europe, Villemoes pointed out the 22 Mbps Unisource IP subnet, the ATM VP subnet and the 34 Mbps leased ATM line between Germany and CERN. Villemoes emphasised that the last mentioned line is indeed part of the project, the only successful surviving element of the original plan to interconnect research networks at 34 Mbps., although the lines radiating to Nordic countries from the Stockholm node will also be at 34 Mbps.
Villemoes said that CERN (and some other partners) will work with both the Unisource and FUDI subnets. He drew attention to the connections towards the east, to Austria, Hungary and, in the future, to the Czech Republic. Three countries have yet to join up formally: Belgium, Luxembourg and Portugal. The time scale is now such that project start-up cannot be before February '97 and is more likely to be around end-March.
He finished by showing a very complex diagram of the physical TEN-34 network topology, pointing out that it is the responsibility of Dante to make IP run over the whole thing.
Opening the discussion, Frese and Valente noted that some practical details concerning payment modalities remain to be fixed.
Hoogland observed the lack of a connection to Poland but Villemoes said that the academic world in that country had not shown interest yet. Hoogland went on to ask about links from TEN-34 to transatlantic lines. Valente replied that the EC programme is divided into several work packages. Package number 6 addresses intercontinental links, at least two of which should be defined. Work is now in progress to collect the wishes of the national research networks. Several are interested. Internal discussions go on but there is strong resistance from one source that is not in favour of working with a unique point of access to the USA. Other national research networks are known to prefer to make their own connection arrangements directly (with IP over ATM). Further to this point, Villemoes noted that TEN-34 has now only 22 Mbps capacity and will be saturated within one year. There would thus be a fundamental mismatch in linking it to transatlantic traffic, where the bandwidth requirements are at least as high. Berger remarked that this seemed to be confirmation that TEN-34 would definitely not satisfy needs for traffic to the USA.
It was noted that CERN will also now benefit from the 40% EC rebate, but May and Williams both emphasised that, despite the number of lines converging on CERN, the actual bandwidth available to CERN as an end-node will only be 7 Mbps, initially at least.
Berger remarked that the countries participating both in TEN-34 and in Europanet would be turning off the latter, although there will be a bridge between the two.
Kalmus said that, in view of the short lifetime of TEN-34, it is now clearly time to concentrate on a follow-on solution. He expressed some disappointment about the final form that the project is taking, and about the fact that liberalisation is not driving down costs more rapidly towards US levels. There were mixed feelings among other members as to whether such price reductions would materialise and Kalmus concluded the point by saying that we must now hope for a better followon project.
Speaking of CERN resources, Foà emphasised that, although spending on networking might have to increase, the total budget was being compressed and so such a rise could only take place at the expense of other areas. He also noted the intention of several countries to cancel their leased lines to CERN, stressing that the fact that CERN's connection to public international networking increases from 2 to 7 Mbps should not necessarily be taken as a sign that leased lines are no longer required.
A rapid poll of members revealed the following picture
for countries that presently have leased lines to CERN:
| Country | ||
| IT | 8 | Stop when TEN-34 operational |
| FR | 2.5 | Will try to keep |
| NL | 2.5 | Stop when TEN-34 operational |
| UK | 0.25 | Stop when TEN-34 operational |
| DE (DESY) | 0.5 | Stop when TEN-34 operational |
The bandwidth being discarded thus largely exceeds the potential gain from TEN-34. Villemoes promised to prepare a complete table of this data.
Summing up, Kalmus said that he was alarmed by the seriousness of the situation. It would clearly be bad for connectivity to decrease. He proposed a three-point plan of action:
1) Find out the total traffic to CERN and circulate this information to members by Email. Williams, however, stated that essentially all the presently available capacity is fully used.
2) Appeal to those organisations financing leased lines to CERN not to switch them off prematurely. Kalmus undertook to do this in the UK and was assured that similar efforts would be made in France and Italy. The situation for the Nordic countries was said to be unclear and it was also not evident how much influence could be brought to bear in Germany.
3) Get TEN-34 usage statistics rapidly when it starts, since any request from the community for CERN to spend more on networking must clearly be properly motivated.
A short note from Devenish has already been distributed to members.
Devenish first reviewed the central computing facilities at RAL, noting that they comprise three platforms linked to a data store. The centre works quite well but is now being reviewed by the CNAP committee of which he is the chairman. The underlying model until now has been that the primary data store should be at an accelerator site. There was now a feeling, however, that this might not be generally valid in the longer term future, and already was not so for some small experiments.
The strategic questions being asked are:
Must a central facility be coupled to a data store?
What is the HEP computing model in the medium term?
What are the responsibilities of the collaborations for off-line computing?
What is the importance of developments in networking on the organisation of central and local computing facilities?
Underlying these questions are the deeper considerations on whether or not regional centres are needed and on the role of national laboratories as centres.
Devenish said that he would like to gather comments on these issues from CERN, DESY, SLAC, FNAL, IN2P3 and Frascati/Gran Sasso.
Williams noted that the ATLAS and CMS computing technical proposals (CTPs) will be ready before the end of '96 and will address many of these issues. If necessary, ATLAS and CMS could be asked to present their CTPs to HEP-CCC and the chairman of the LCB committee could also be invited. Valente added that no final conclusions are possible at the moment and that all collaborations are actively discussing the matter.
Kalmus said he felt that HEP-CCC cannot usefully address the question of LHC computing models until after the CNAP report and after the thinking of the collaborations is more clear. He emphasised that the work of CNAP in this affair is to be viewed in a purely UK domestic context and should not be generalised.
A copy of Frese's and Gaines' overheads has been distributed to members.
Frese stated clearly that the idea of a transatlantic pilot project using the CANTAT cable is dead. Instead, DFN and Deutches Telekom have introduced what is essentially an extension of B-WiN to a point-of-presence at Washington DC, where there is good connectivity to MCI. The European end is at Frankfurt. The capacity is 2*45 Mbps (on 155 Mbps lines) in the TAT 12 and TAT 13 cables. The link uses routers interconnected by circuits emulated on ATM. Frese stressed that there is no through-service of native ATM due to regulatory problems in the USA.
Gaines spoke of activity within ESnet and began by reviewing the ESnet connectivity within the USA, going on to list the transatlantic links presently used by HEP researchers - to Italy at 1.5 Mbps, to Germany at 1.5 Mbps and to CERN at 2 Mbps. In addition, usage is made of a number other general use links, which have much higher total bandwidth.
He went on to speak of the Internet II project which involves a consortium of 35 universities and has the goal of studying a next-generation Internet. Each partner is contributing 500 kUSD/year and federal funding is also being requested. It is expected that the collaboration will grow in size and will also involve commercial computer and telecoms suppliers. Gaines emphasised that the goal is not to design a future public Internet but rather a general purpose academic network. He also mentioned the Clinton networking initiative, which aims to connect universities and national labs with a bandwidth 100-1000 times greater than today's Internet. The likely investment is 100 MUSD over the next years.
Finally, Gaines spoke of an initiative at Berkeley for the support of collaboration and resource sharing. This is focused on studio-based video-conferencing which is heavily used and felt to be very effective. Studio scheduling is managed via a convenient Web application. The service is presently being expanded to 40 ports with a PictureTel MCU.
May asked about plans to give good CERN connectivity to US physicists in LHC collaborations. Gaines replied that it is recognised that network funding needs to be ramped up but the technical details are still unclear. The bandwidth requirements for the short-term future are not clear either and in fact the main bandwidth use at present is for video-conferencing. Repellin noted that France has foreseen spending on networking on top of the ATLAS and CMS budgets, and he hoped that the same would be done in the USA.
Kalmus queried the place of the next generation Internet work in the HEP community and Gaines agreed that it would very likely be better for it to be done elsewhere. Villemoes, on the other hand, drew attention to the arguments that are put forward to the effect that the development emphasis will be different if the work is left to the commercial operators.
Valente drew attention to the fact that many US HEP institutes have very poor connectivity to the ESnet backbone, if at all. Some US collaboration data, for example that at Urbana, is simply inaccessible from Europe.
A copy of Shiers' overheads has been distributed to members.
Shiers began by noting that his presentation would focus on LHC++ and would concentrate on the effect on outside laboratories and universities. The licensing model being pursued for LHC++ seeks to minimise licensing costs, particularly for outside institutes, while also being as simple as possible.
He emphasised that, despite the popularly held view, the provision of the present CERN program library, CERNlib, is not without cost. It has long contained commercial components and has taken at least 300 man-years of HEP effort to develop.
With the new generation of running experiments, such as NA48, CERNlib is reaching its limits and it clearly does not have the functionality needed for LHC. The effort needed to produce a replacement that is adequate for LHC will be very considerable, of the order of 500 man-years, and the time available is short since something must certainly be available within 3 years at the most.
The option of developing all the necessary software in the community is simply not viable given the scale of the problem and the necessary time scale. For this reason, LHC++ is seeking to use industry standard solutions wherever possible, concentrating the HEP effort on the HEP-specific parts. This approach brings the added benefits that standard commercial products are much cheaper, when development and maintenance effort are taken into account, and are also normally better since they are exercised by a large community and are well supported and documented.
LHC++ strategy is thus based on a commercial infrastructure of graphics and visualisation software, maths libraries, object database and C++ standard libraries, to which the HEP-specific code is being added, both general and experiment-specific.
This approach implies of course the acquisition of the necessary licences for the commercial components. Most of the licences needed for LHC++ use on the CERN site have already been acquired. For off-site use, licence negotiation is more difficult since the conditions are hard to define. HEP-wide licensing sounds attractive but is normally impossible to negotiate. Site-wide (e.g. an institute) or project-wide (e.g. the LHC programme) licensing is much easier.
Shiers drew attention to the need to distinguish between developer licences and runtime licences. The former are normally rather expensive, while the latter are cheap or may even be free. He emphasised that runtime licences allow not only the execution of pre-built modules but also relinking along with user code. This state of affairs thus encourages clear thinking about the number of people who should be actively engaged in code development.
In considering the implications of licensing costs, one has to take into account the needs of the smaller institutes as well as the larger ones and also understand the implications for places where money is scarce, while at the same time staying within the bounds of affordability for CERN.
With these considerations in mind, an LHC++ licensing model is proposed in which CERN buys site-wide runtime licences and sufficient floating developer licences for all the CERN reference platforms. Outside institutes should buy the developer licences that they need. Runtime licences should be bought centrally by the collaborations or eventually by the major institutes. Where appropriate, CERN is willing to fight for prices on HEP's behalf and to act as a distribution centre for documentation and software.
This model minimises the costs to the outside institutes since, for use with the CERN programme, they pay nothing for runtime licences. Developer licences could also be covered centrally by the collaborations although they may wish contributions from the institutes. Shiers additionally suggested the consideration of development platforms where the developer licences are free (e.g. OpenGL on Windows NT).
For non-CERN work, the major institutes can still obtain consistent HEP-wide prices. Non-HEP academic use is clearly not covered by this scheme in the same way that it was for CERNlib but universities can typically obtain much higher academic discounts than are available to HEP institutes.
Overall, Shiers estimated the licence acquisition cost for the commercial parts of LHC++ per LHC experiment as being some 50 kUSD per year - considerable bargain for so much software. For comparison, he noted that the 1996 collaboration costs to CERN for development of the HEP-specific CLHEP class library had alone been twice the cost to an LHC experiment of all the graphics licences. At the level of the community, the ratio was four times. This was without costing the manpower for the actual software development.
Concluding, Shiers said that there appears to be an affordable model for LHC++ licensing. The licences needed initially on the CERN site have been acquired and there are offers to extend these licences to outside institutes. What remains to be tackled are agreements with the collaborations on the funding of runtime licences for external use.
May and Hoogland expressed some surprise that the costs of licensing appeared to be so low. Shiers stressed that he had only been talking of the LHC++ components and that the prices mentioned are not yet contractual. Speaking of the Objectivity OODBMS component, he said that this is cheaper than Oracle because negotiations had started at the right time in the product cycle and because the market is different. In addition, the cost is spread over the experiments and over ten years. Overall, however, the real point is that all this software would cost the community very much more to write itself. At the same time, he recognised the need for constant vigilance to make sure that the model remains affordable. He said that maintenance is typically 10-15% of the purchase price and noted that the software maintenance costs in CERN are not increasing.
Williams noted that, although Objectivity is an important part, there are many other components as well and it is important for HEP-CCC to understand the impact of the move to the OO paradigm on spending for software licences.
Devenish wondered if there is a risk that people working in multiple experimental programmes end up making multiple payments. Shiers replied that this need not be the case if things are organised properly and added that the list of products in LHC++ was being evolved in consensus with the community.
Kasemann worried about the difficulty of guaranteeing CERN prices to outside institutes but Shiers said that two vendors had already promised to do this. All that is required is the identification of bona fide HEP users. Williams added that outside institutes can sometimes get even better deals than CERN.
Concluding this item, Kalmus stressed the importance that the whole community be kept informed of the licensing negotiations taking place, either via the membership of the LHC++ working group or otherwise. He thanked Shiers and CERN for the initiative taken, saying that it appeared to him evident that the strategy of making use of commercial products wherever appropriate is the only one possible, the effort to write everything within HEP simply being unavailable. The community must therefore face up to the necessity of establishing budget lines for software purchase and maintenance.
Williams overheads are available at http://nicewww.cern.ch/~davidw/public/icfa.htm (also linked to from the HEP-CCC home page).
Since time was running short, Williams confined himself to noting that ICFA presently has two working groups, one on detectors and one on other problems. It is now proposed to set up another on international networking and to use it as an aid to collaborative working in the community. It should be formed officially at the next ICFA meeting in January '97.
A copy of Albrecht's overheads has been distributed to members.
Albrecht began by noting that the architecture for HERA-B computing is not yet finalised but that boundary conditions have been set. The target is to have the software ready in '98.
The physics aims of the experiment are principally in the area of CP violation but B spectroscopy is also of interest. A fixed target is used, consisting of eight transverse wires, spaced along the beam direction. This arrangement is expected to result in some five interactions per beam bunch (at 10 MHz) but it is hoped that the spatial separation and localisation of the vertices will help the reconstruction task which is rendered rather difficult by the high multiplicity of secondaries (roughly 200 in front of the ECAL). The worst case occupancy of the outer tracker can reach 25%. It is clear these numbers tend in the direction of those expected in LHC experiments. The expected physics output rate for the main B0 ® J/Y channel is only 0.04 Hz, with other triggers contributing 0.4 Hz.
Triggering will reduce the 10 MHz raw rate in four stages:
1 Hardware track finding in a few layers (Þ 50 kHz).
2+3 The distinction is not yet decided on but 2 will be linked with the data acquisition and 3 with reconstruction and between them they will involve a track fit in all layers, a vertex fit and the calculation of an impact parameter with respect to the target wires (Þ 100 Hz).
4 Full on-line reconstruction (Þ 20 Hz overall - 0.5 Hz physics).
Trigger levels 3 and 4 are seen as being handled by an on-line farm of more than 100 processors. No distinction is seen between the software run in this farm at that used off-line. Ideally, the complete and final reconstruction should be done on-line with off-line reprocessing being only needed in exceptional circumstances. This mode of operation requires, of course, the availability of calibration constants on-line, even for the operation of the trigger. This is a significant challenge. Since maintaining separate capacity elsewhere for reprocessing is seen as too expensive, the intention is to use the on-line farm for this task during shutdowns.
Speaking in more detail of the calibration, Albrecht said that this is essential from trigger level 2 onwards. A continuous monitoring of constants will be required, with continuous updating to follow slow trends and suitable alarms should sudden jumps be detected. Such alarms would result in the running of special calibration and alignment tools as necessary.
The whole system is seen as placing enormous demands on manpower but the collaboration is not in a position to quantify these at present.
Assuming a 20 Hz recorded data rate at 100 kbytes/event, it is expected that there will be a need for 20 Tbytes/year of mass storage for DSTs and simulation data, along with a further Tbyte/year of disk for data that is being actively analysed. It is intended to keep all the data at DESY. At this point Williams remarked on the apparent anomaly that BaBar, with events a quarter of the size of HERA-B (but 5 times the data rate), is estimating a need for 85 Tbytes/year of mass storage, also including simulation data (see e.g. CCC/CERN/96/03).
Talking of computers for analysis, Albrecht said that the type had still be decided, as well as the operating system (Linux or Windows NT).
Software distribution is now being managed under CVS, the conversion having caused no problems and many advantages becoming apparent. Remote use of CVS is already working well between Zeuthen and Hamburg. The only point noted was that CVS requires write privileges even when read-only would suffice - a potential safety consideration. There is a universal software framework called ARTE used both on-line and off-line. Under this, developers are urged to use the same utilities, the same I/O and the same data structures. The benefits include a large independence from programming language. The data model is borrowed from ADAMO and the memory management is a restricted form of ZEBRA. Interface routines ensure that the ZEBRA structures are accessible not only from FORTRAN but also from C and C++. The data structures themselves are defined in a way that is language-independent, with the ARTE-DD tool deriving the necessary bindings to the various languages. C++ has been declared to be the preferred language but in practice most existing code is in FORTRAN, with the new code being written in C++.
The strategy for reconstruction is to start with pattern recognition on both sides of the magnet and then proceed to tracking through the magnet. At the moment an efficiency of about 95% is reached with simulated data and this must clearly be increased. Work is also needed on the processing time, which is currently five times more than could be handled by a farm of the proposed size. Vertex reconstruction, adapted from ALEPH, is satisfactory but particle identification is still being worked on.
Summarising, Albrecht pointed out that the computing for HERA/B, in terms of CPU, storage and networking, does not involve too large an extrapolation from present experiments, being within a factor of two for all aspects. The goal of on-line reconstruction does, however, place large demands on software reliability. The language-independence of the data structures is felt to be a strong point. Test data is already beginning to come in and the collaboration is looking forward to taking data with the complete detector in 1998.
Gaines remarked that, despite the evident need for software reliability, the collaboration did not seem to be placing any special emphasis on techniques for enhancing reliability. Albrecht conceded that this is so, apart from the recommendation that new software should be in C++. He felt that the risks would be diminished by the fact that the development team is small. Out of a collaboration membership of 220, only some 10 man-years/year spread over 30-40 people is going into software, less than 5% of the total effort. Development is also geographically concentrated at Zeuthen and Hamburg.
Other members continued to express worries about the lack of resources to enhance software reliability and Albrecht said that the DESY Management had been made aware of the problem.
Kalmus thanked Albrecht for his interesting presentation and wished the collaboration good luck.
11.1 Kalmus reminded members that his term as chairman would soon come to an end and he requested that suggestions as to possible successors should be Emailed to him urgently. He hoped that a clear consensus would emerge, otherwise he would report the situation to the committee and ask for advice.
11.2 Kalmus noted that this was the last meeting for David Williams as a member since his mandate as CN Division Leader comes to an end in December '96. He had no doubt, however, that Williams would be periodically returning to future meetings with reports. He thanked Williams for the immense amount of work that he had invested in HEP-CCC matters, considerably easing the job of the chairman, and also for his helpfulness in making sure that the committee's recommendations were passed on and acted upon.
Replying, Williams thanked the whole committee for its collaboration and good advice, and asked members to accord to his successor, Juergen May, all possible support in his difficult task of integrating computing at CERN.
The next meeting of HEP-CCC will take place at CERN on 18 April 1997.
Action: Next Meeting
The date of the next meeting was fixed to be Friday 18 April 1997, at CERN. Status: new.
Action : Secretary, all members
The 1997 summer meeting will take the form of a 2-day meeting at RAL (fixed subsequently to be 89 July - Sec.)
Action: 1997 Summer Meeting
The dates of the 1997 summer meeting were fixed to be Tuesday and Wednesday 8-9 July 1997, at RAL. Status: new.
Action : Secretary, all members
None.
The following documents related to this meeting have already been distributed:
Transparencies
Sets of papers/overheads that were distributed at the meeting
Others
D. Jacobs
37. 2/96. Update on status of the TEN-34 project. P. Villemoes is asked to talk again about progress with the TEN-34 project at the 18/4/97 meeting. Status: ongoing, kept on future agenda list.
Action : P. Villemoes, Secretary
38. 2/96. Use of groupware tools by RD44. RD44 will be asked to make a presentation on its use of groupware tools for software development at a future meeting It was proposed to schedule this item at the 18/4/97 meeting. Status: pending, retained on the future agenda list.
Action : Secretary
39. 2/96. Tools for distributed editing of papers. HTASC is asked to survey currently available tools for distributed editing of papers and future prospects for this application. A report will not be available before the 18/4/97 meeting. Status: ongoing.
Action : HTASC
41. 6/96. Gather information on 4th Framework project applications. Ask P.G. Innocenti to assemble a spreadsheet of information on EU 4th Framework project applications from HEP (including the unsuccessful ones), make it available on the Web and invite (via HTASC) further entries from around Europe. Ask P.G. Innocenti to address the 18/4/97 meeting on this topic, if possible providing written information earlier. Status: ongoing, transferred to Secretary.
Action : Secretary
44. 6/96. INFN plans for EDMS. Ruggieri to inform HEP-CCC about plans in INFN concerning EDMS when some progress has been made in this direction. Status: ongoing.
Action : Ruggieri
45. 6/96. CEDAR update. Invite C. Hauviller to present an update on the CEDAR project at the first HEP-CCC meeting in 1997 (18/4). Status: ongoing, retained on future agenda list.
Action : Secretary
47. 11/96 Proposals for HEP-NT Chairman. Kasemann is asked to come to the April '97 HEP-CCC meeting with names to propose as HEP-NT chairman. Status: new.
Action : Kasemann
1. 2/95. Network Security. It was proposed to let the HTASC prepare this topic before discussion in HEP-CCC. Status : pending - referred to HTASC.
Action : Chairman, HTASC
2. 9/94. Mark-up Languages. Gaemers proposed an agenda item on Mark-up Languages (such as TeX, LaTeX and SGML and their futures for a future meeting. It was decided at the 10/95 meeting that it would still be interesting to get a report on the topic at some time if a suitable speaker could be found. Status : pending.
Action : Secretary, Chairman
3. 2/95. Reports from Experiments. Williams
and others proposed that NA49 and also the DESY experiments should,
over time, be presented to the HEP-CCC. A report from KLOE at
DAPHNE was also proposed. (NA48 discussed at 10/95 meeting; KLOE
discussed at 2/96 meeting, BaBar discussed at 6/96 meeting and
HERA-B discussed at this meeting).
Status : updated, ongoing.
Action : Secretary, Chairman
4. 2/95. North American Issues. The North American Observers are invited to propose one or two issues concerning computing in their region which could be presented at each meeting. Status: ongoing.
Action : I. Gaines, L. Price
10. 6/95. Annual progress reports from the LHC experiments (at the meeting away from CERN?). Status: ongoing.
Action : Secretary
11. 6/95. Monitor the international networking situation. HEP-CCC will continue to monitor regularly the international networking situation in order to try to arrive at a better understanding. Status: ongoing.
Action : Secretary
12. 10/95. Reports from the HTASC. A report on the work of the HTASC will be a regular feature of future meetings. Status: ongoing.
Action : M. Kasemann, Secretary
15. 2/96. Update on status of the TEN-34 project. P. Villemoes is asked to talk again about progress with the TEN-34 project at the 18/4/97 meeting. Status: ongoing.
Action : P. Villemoes, Secretary
16. 2/96. Use of groupware tools by RD44. RD44 will be asked to make a presentation on its use of groupware tools for software development at a future meeting. It was proposed to schedule this item at the 18/4/97 meeting. Status: pending.
Action : Secretary
17. 2/96. Windows95/NICE at CERN. A presentation on this topic was in the end not requested for this meeting due to lack of agenda time. Retain for 1997? Status: ongoing.
Action : Secretary
22. 6/96. CEDAR update. Invite C. Hauviller to present an update on the CEDAR project at the first HEP-CCC meeting in 1997 (18/4). Status: ongoing.
Action : Secretary
23. 11/96. Harmonisation of HEP proposals for the European Commission Training and Mobility of Researchers programme. Invite P.G. Innocenti to talk on this at the first HEP-CCC meeting in 1997 (18/4). Status: new.
Action : Secretary
24. 11/96. Annual reports from FOCUS. HEP-CCC should receive regular annual reports from FOCUS, normally for the summer meeting (June/July). Status: new.
Action : Secretary
D. Jacobs