[CI] RFC: Major reorg of Cluster Tools

Discussion:

Brian J. Watson

2003-05-21 19:39:04 UTC

To the CI and OpenSSI communities-

The CI and OpenSSI user-mode code has been thrown together in an ad hoc
fashion since the beginning of these projects. It has been lumped
together into one RPM, source release, and CVS module that supports two
projects, two Linux distros, and three architectures. Its structure is
inconsistent and confusing to novice users and developers, forbids the
replacement of commands owned by base RPMs, and encumbers the production
of releases.

I propose the following changes to clean-up things:

Binary release changes
----------------------
* roughly separate cluster-tools into its CI and OpenSSI components
* roll just the CI part as the cluster-tools RPM
* spin off the OpenSSI part as an openssi-tools RPM and a set of
enhanced base RPMs, such as mount (util-linux), devfsd, nfs-utils, and
ipvsadm (ha-lvs)
* include kernel-openssi, cluster-tools, openssi-tools, the enhanced
base packages, documentation, and a simple installation script in a new
OpenSSI binary release
* eventually roll a CI binary release containing kernel-ci and cluster-tools
* libcluster should retain OpenSSI support in the reduced cluster-tools,
since it's not worth the effort to separate this code

Source release changes
----------------------
* combine the stripped-down cluster-tools with the CI kernel code to
make a new CI source release
* continue to include combined CI and OpenSSI code in OpenSSI releases
* eliminate Cluster Tools as a separate source release
* organize the CI and OpenSSI source trees to reflect the binary packages
* organize the large number of commands for openssi-tools into
functional groups

CVS repository changes
----------------------
* reorganize the repository to match the new source trees
* adjust the system of Makefiles to reflect the new source organization
* exclusively adopt Aneesh's GNU build system, so that only one set of
Makefiles needs to be reorg'd
* use a consistent system for supporting multiple distributions and
architectures
* write scripts for automatically producing source and binary releases

This is how I propose laying out the CI repository:

ci
|-- AUTHORS (was in cluster-tools)
|-- COPYING
|-- ChangeLog
|-- Makefile (rules for rolling releases)
|-- Makefile.am (rules for building and installing)
|-- NEWS (was in cluster-tools)
|-- README
|-- cluster-tools
| |-- Makefile.am
| |-- cmd/ (moved some commands to openssi-tools)
| |-- libcluster/
| `-- man/ (moved some pages to openssi-tools)
|-- configure.ac (was in cluster-tools)
|-- doc/
|-- kernel/ (was ci-linux/ci-kernel)
|-- kernel.configs
| `-- config.i586 (was in ci-linux/ci-kernel)
|-- kernel.patches (was ci-linux/3rd-party)
| |-- common/
| `-- i386/
`-- specs
|-- cluster-tools.spec (was in binary/rh7.3)
`-- kernel.spec

This is how I propose laying out the OpenSSI repository:

openssi
|-- AUTHORS
|-- COPYING
|-- ChangeLog
|-- Makefile (rules for rolling releases)
|-- Makefile.am (rules for building and installing)
|-- NEWS
|-- README
|-- configure.ac
|-- devfsd/ (was in cluster-tools/ssi)
|-- distro-pkgs (distro-specific packages)
| |-- debian (?? not sure about Debian pkg names ??)
| | |-- init-scripts (was cluster-tools/debian-init-scripts)
| | | |-- S01devfsd
| | | |-- S10checkroot.sh
| | | |-- S35mountall.sh
| | | |-- hostname.sh
| | | |-- install-deb-initscripts (was in cluster-tools)
| | | `-- rcSSI
| | `-- specs
| | `-- init-scripts.spec
| `-- redhat
| `-- specs/
|-- doc
| |-- INSTALL (was in binary/rh7.3)
| |-- INSTALL.cvs
| |-- INSTALL.gfs
| |-- INSTALL.ia64
| |-- INSTALL.pxe (was in cluster-tools)
| |-- INSTALL.source (was ssic-linux/doc/INSTALL)
| |-- README-CI
| |-- README-mosixll
| |-- README.cfs
| |-- README.hardmounts
| `-- cdsl
|-- ipvsadm/ (was cluster-tools/ha-lvs)
|-- kernel/ (was ssic-linux/ssi-kernel)
|-- kernel.configs
| |-- config.alpha (was in ssic-linux/ssi-kernel)
| |-- config.i586 (was in ssic-linux/ssi-kernel)
| |-- config.ia64 (was in ssic-linux/ssi-kernel)
| |-- config.ia642 (was in ssic-linux/ssi-kernel)
| |-- config.uml (was in ssic-linux/ssi-kernel)
| `-- kernel-2.4.18-i586-ssi.config (was in binary/rh7.3)
|-- kernel.patches (was ssic-linux/3rd-party)
| |-- common/
| |-- i386/
| `-- ia64/
|-- nfs-utils/ (patched by cluster-tools/ssi/nfslock.patch)
|-- openssi-tools
| |-- Makefile.am
| |-- arch
| | |-- Makefile.am
| | |-- alpha
| | | |-- Makefile.am
| | | `-- ssi_arch.pm (was cluster-tools/ssi/ssi_alpha.pm)
| | |-- i386
| | | |-- Makefile.am
| | | `-- ssi_arch.pm (was cluster-tools/ssi/ssi_i386.pm)
| | |-- ia64
| | | |-- Makefile.am
| | | `-- ssi_arch.pm (was cluster-tools/ssi/ssi_ia64.pm)
| | `-- uml
| | | |-- Makefile.am
| | |-- clustertab (was in cluster-tools/ssi)
| | `-- fstab (was in cluster-tools/ssi)
| |-- distro
| | |-- Makefile.am
| | |-- debian
| | | |-- Makefile.am
| | | |-- inittab.ssi (was cluster-tools/ssi/inittab.ssi.debian)
| | | `-- ssi_distro.pm (was cluster-tools/ssi/ssi_debian.pm)
| | `-- redhat
| | |-- Makefile.am
| | |-- inittab.ssi (was cluster-tools/ssi/inittab.ssi.redhat)
| | `-- ssi_distro.pm (was cluster-tools/ssi/ssi_redhat.pm)
| |-- fs
| | |-- Makefile.am
| | |-- cfs_mount.c (was in cluster-tools/cmd)
| | |-- cfs_remount.c (was in cluster-tools/cmd)
| | |-- cfs_setroot.c (was in cluster-tools/cmd)
| | |-- ckroot.ssi (was in cluster-tools/cmd)
| | |-- cmount.c (was in cluster-tools/cmd)
| | |-- gfscf.cf (was in cluster-tools/ssi)
| | |-- mount_remote_root.c (was in cluster-tools/cmd)
| | |-- pool0.cf (was in cluster-tools/ssi)
| | `-- pool0cidev.cf (was in cluster-tools/ssi)
| |-- init.ssi/ (was cluster-tools/ssi/init)
| |-- initscripts
| | |-- Makefile.am
| | |-- rc.modules (was in cluster-tools/ssi)
| | |-- rc.nodedown (was in cluster-tools/ssi)
| | |-- rc.nodeup (was in cluster-tools/ssi)
| | |-- rc.sysinit.nodeup (was in cluster-tools/ssi)
| | `-- rc.sysrecover (was in cluster-tools/ssi)
| |-- keepalive/ (was in cluster-tools/ssi)
| |-- man
| | |-- Makefile.am
| | |-- keepalive.1M.html (was in cluster-tools/man)
| | `-- spawndaemon.1M.html (was in cluster-tools/man)
| |-- net
| | |-- Makefile.am
| | |-- node_hostname.c (was in cluster-tools/cmd)
| | `-- setport_weight.c (was in cluster-tools/cmd)
| |-- proc
| | |-- Makefile.am
| | |-- load_level_com.h (was in cluster-tools/cmd)
| | |-- loadlevel.c (was in cluster-tools/cmd)
| | |-- loads.c (was in cluster-tools/cmd)
| | |-- migrate.c (was in cluster-tools/ssi/migrate)
| | |-- onall.c (was in cluster-tools/cmd)
| | |-- onnode.c (was in cluster-tools/cmd)
| | `-- where_pid.c (was in cluster-tools/cmd)
| `-- sysadmin
| |-- Clustertab.pm (was in cluster-tools/ssi)
| |-- Makefile.am
| |-- addnode (was in cluster-tools/ssi)
| |-- addnode.dev (was in cluster-tools/ssi)
| |-- addnode.pm (was in cluster-tools/ssi)
| |-- chnode (was in cluster-tools/ssi)
| |-- cluster_lilo (was in cluster-tools/ssi)
| |-- cluster_mkinitrd (was in cluster-tools/ssi)
| |-- mkdhcpd.conf (was in cluster-tools/ssi)
| |-- openssi_addnode (was in cluster-tools/ssi)
| `-- openssi_cluster_create (was in cluster-tools/ssi)
|-- specs
| |-- devfsd.spec
| |-- ipvsadm.spec
| |-- kernel.spec (was binary/rh7.3/kernel-2.4.spec)
| |-- nfs-utils.spec
| |-- openssi-tools.spec
| `-- util-linux.spec
`-- util-linux/ (was in cluster-tools/ssi)

Let me know what you think,

Brian

Aneesh Kumar K.V

2003-05-22 02:25:09 UTC

Permalink

Brian,

On Thu, 2003-05-22 at 03:07, Brian J. Watson wrote:

[...snip....]

You may want to wait for the changes done yesterday by laura to be
merged back to the trunk before reorganising the CVS. Also cluster-tools
patch from Scott also need to be not applied.

It would be better if we wait for a day or so and say no more submits
and then reorganise.Then we can get the reorganised RH branch and Trunk
built and then open up the CVS for further submit.

Post by Brian J. Watson
ci
|-- AUTHORS (was in cluster-tools)
|-- COPYING
|-- ChangeLog

How are you going to organise the Changelog .I mean what about the
previous contents. ?

install-deb-initscripts is not needed this job is now done by
openssi_cluster_create.

I guess init-script.spec is going to be script which will help in
building debian packages.

Can't we drop addnode and addnode.dev and use openssi_cluster_create
and openssi_addnode

Great!!!!.

-aneesh

Aneesh Kumar K.V

2003-05-22 03:07:07 UTC

Permalink

Post by Aneesh Kumar K.V
You may want to wait for the changes done yesterday by laura to be
merged back to the trunk before reorganising the CVS. Also cluster-tools
patch from Scott also need to be not applied.

That should be read as patch from Scott NEED to be Applied.

-aneesh

Aneesh Kumar K.V

2003-05-22 03:10:09 UTC

Permalink

That should be read as patch from Scott NEED to be Applied.

-aneesh

Brian J. Watson

2003-05-23 03:20:02 UTC

Permalink

Post by Aneesh Kumar K.V
It would be better if we wait for a day or so and say no more submits
and then reorganise.Then we can get the reorganised RH branch and Trunk
built and then open up the CVS for further submit.

That's my plan. I'm giving a few days for comments, then I'm going to
ask everyone to refrain from checkins while I do the reorg. Then I'll
help developers adapt their sandboxes to the new structure.

Post by Aneesh Kumar K.V
How are you going to organise the Changelog .I mean what about the
previous contents. ?

I'm thinking of ditching the old changes and starting afresh. The
history of cluster-tools and ci-linux as separate releases is not that
interesting going forward.

Post by Aneesh Kumar K.V
install-deb-initscripts is not needed this job is now done by
openssi_cluster_create.

Okay. I'll eliminate it.

Post by Aneesh Kumar K.V

I guess init-script.spec is going to be script which will help in
building debian packages.

In RPM-land, a .spec file provides instructions on how to build the
binaries for a package, specifies dependencies on other packages,
includes a description of what the package does, etc. Debian might use
something different to build its own packages. I'm not sure what.

Post by Aneesh Kumar K.V
Can't we drop addnode and addnode.dev and use openssi_cluster_create
and openssi_addnode

Not yet. I want to play with them first, make some comments, and see
what needs to be changed in the documentation. We can cross this bridge
after finishing the reorg.

BTW, Scott's change to add PXE support will help me do this. It's been
awhile since I've used a test cluster that can do Etherboot, so I
haven't even played with the original addnode for a long time. :(

Post by Aneesh Kumar K.V

Post by Brian J. Watson
Let me know what you think,

Great!!!!.

Thanks!!

Brian

Aneesh Kumar K.V

2003-05-23 03:34:08 UTC

Permalink

Post by Brian J. Watson

That's my plan. I'm giving a few days for comments, then I'm going to
ask everyone to refrain from checkins while I do the reorg. Then I'll
help developers adapt their sandboxes to the new structure.

Post by Aneesh Kumar K.V
How are you going to organise the Changelog .I mean what about the
previous contents. ?

I'm thinking of ditching the old changes and starting afresh. The
history of cluster-tools and ci-linux as separate releases is not that
interesting going forward.

Then i would request to start with fresh ChangeLogs with new formats for
both the repository and old Changelogs can be renamed.

ie in the SSI repository we will have
ChangeLog and ChangeLog.0( This is the old Changelog ). ChangeLog will
have the first line as previous cluster-tools ChangeLog can be found at
so and so place

in the ci we will have ChangeLog,ChangeLog.0( this is ci-linux change
log ) and ChangeLog.cluster-tools( this is cluster-tools changelog ) and
we will have a new entry in ChangeLog saying the old cluster-tools
ChangeLog can be found in ChangeLog.cluster-tools.

I would also say that the ChangeLog fromat can be

DD/MM/YYYY username <useremailaddress>
* Changes

see this .
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/opendlm/opendlm/ChangeLog?rev=HEAD&content-type=text/plain

I found this format pretty useful when tracking bugs.

Post by Brian J. Watson

Post by Aneesh Kumar K.V
install-deb-initscripts is not needed this job is now done by
openssi_cluster_create.

-aneesh

Brian J. Watson

2003-06-02 22:25:07 UTC

Permalink

Post by Aneesh Kumar K.V
Then i would request to start with fresh ChangeLogs with new formats for
both the repository and old Changelogs can be renamed.
ie in the SSI repository we will have
ChangeLog and ChangeLog.0( This is the old Changelog ). ChangeLog will
have the first line as previous cluster-tools ChangeLog can be found at
so and so place
in the ci we will have ChangeLog,ChangeLog.0( this is ci-linux change
log ) and ChangeLog.cluster-tools( this is cluster-tools changelog ) and
we will have a new entry in ChangeLog saying the old cluster-tools
ChangeLog can be found in ChangeLog.cluster-tools.
I would also say that the ChangeLog fromat can be
DD/MM/YYYY username <useremailaddress>
* Changes
see this .
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/*checkout*/opendlm/opendlm/ChangeLog?rev=HEAD&content-type=text/plain
I found this format pretty useful when tracking bugs.

I agree with your suggestions. I would also like to insert the following
line into the ChangeLog every time I roll a release:

DD/MM/YYYY CI x.x.x
-- or --
DD/MM/YYYY OpenSSI x.x.x

Sound okay?

Brian

Aneesh Kumar K.V

2003-06-03 01:33:03 UTC

Permalink

Post by Brian J. Watson
I agree with your suggestions. I would also like to insert the following
DD/MM/YYYY CI x.x.x
-- or --
DD/MM/YYYY OpenSSI x.x.x
Sound okay?

Great !!!

-aneesh

Brian J. Watson

2003-06-03 21:56:06 UTC

Permalink

Post by Brian J. Watson
I agree with your suggestions. I would also like to insert the following
DD/MM/YYYY CI x.x.x
-- or --
DD/MM/YYYY OpenSSI x.x.x

One other suggestion is that we stop including CI changes in the OpenSSI
ChangeLog. When I roll OpenSSI releases, I'll just copy in the CI
ChangeLog as ChangeLog.ci. We don't need to have the two integrated
together.

Brian

Brian J. Watson

2003-06-02 22:34:03 UTC

Permalink

To all developers-

It's time to implement this reorg. Please don't check anything into the
repositories. After I finish updating them, I can help you adapt your
sandboxes to the new code organization.

Aneesh-

Can you help me adapt the configure.ac and Makefile.am files? I'll
checkin a first cut at them today, but you're more experienced than I at
getting them right. :)

-Brian

Aneesh Kumar K.V

2003-06-03 01:32:02 UTC

Permalink

Post by Brian J. Watson
Aneesh-
Can you help me adapt the configure.ac and Makefile.am files? I'll
checkin a first cut at them today, but you're more experienced than I at
getting them right. :)

I will take care of Makefile.am and configure.ac changes.

-aneesh

Brian J. Watson

2003-06-03 04:53:05 UTC

Permalink

Post by Brian J. Watson
I agree with your suggestions. I would also like to insert the following
DD/MM/YYYY CI x.x.x
-- or --
DD/MM/YYYY OpenSSI x.x.x

May I suggest a less internationally ambiguous format: YYYY/MM/DD.
Also complies with ANSI and ISO standards.

I was thinking that myself. What do you think, Aneesh?

Brian

Aneesh Kumar K.V

2003-06-03 05:07:08 UTC

Permalink

On Tue, 2003-06-03 at 12:20, Brian J. Watson wrote:

[..snip..]

Post by Brian J. Watson
I was thinking that myself. What do you think, Aneesh?

Fine. We just need to capture the date of change.

-aneesh