Electronic Data Interchange {EDI}

This covers any form of computer readable data transfer.

A working partnership, consists of :-

For more info see also my old notes on EDI. These date from 1995 but are still relevant.

RTedi

This is a program that shares use of a number of script libraries user to support normal users connecting to roadrunner.

It provides the control and checks needed to run custom import and export applications, under control of a selected user, or the system cron daemon. while preventing execution during system maintenance, and backups.

It also where files are involved in the transfer, maintain history directories, of past transactions.

Transport Protocols

How the data is transferred. There are an almost unlimited number of variations.

EDIFACT

Applications, provided to work with bureau services, complying with the EDIFACT standards, can usually be configured to drop received transaction messages as fixed format file in a convenient directory.

Security is not normally a problem, the only external connection is to the bureau service.

Either configure cron to run RTedi -i shortly after the downloads, from the service, add RTedi -i to a post download event handler.

Roadtech edi send/edi daemon

Custom TCP/IP protocol for transferring information to Roadrunner Consists of a client written in C, a server also written in C, and extensions in the COBOL Run time.

Transfers line delimited data records to the roadrunner server using a tcp connection to an agreed port {5001}. As only a single inbound connection is required it can be easily routed through a firewall even if using NAT.

The daemon prepends the IP4 address of the client to the data records as they are received. There is never any data back to the client other than the ack's for the data received.

FTP

This is one of the oldest protocols on the Internet. The first reference I know of to this was RFC 171 This dates from 23 June 1971, and was superseded by RFC 238 in September 1971.

It is very useful, for use by trusted individuals, between computers on the same network. FTP uses two TCP connections usually a control connection from the client port>1023 to the server port=21, and then one or more data connections from the server port=20 to the client port>1023.

From the security point of view, having two connection in opposite directions, makes firewalling harder. Also the FTP protocol allows directory browsing, to any directory that the user would have had access to using the same name and password from a terminal or telnet connection.

HTTP GET

Some time back I added functions to allow Roadrunner programs to fetch pages/files from a web server. These are available to programmers developing EDI routines, they fetch data using a single out bound connection, which is just about ideal from the security viewpoint.

IBM WebSphere MQ

To send, program calls MQPUT to put messages on to a MQ queue

To receive the program polls receive queue with MQGET

Requires RMcobol v11+ run time, and our MQ extensions. Currently only tested for AIX

Data Formats

How the message is formatted.

Fixed width fields

Data records are line delimited. Each field has a fixed width. If each record starts with record type/and version as in EDIFACT, lines may be of different length.

Comma Separated Variable CSV

Data records are line delimited, data fields are delimited with a character usually a ",".

If the character that is used to delimit the fields may appear in any field, it must be quoted in some way. This may complicate the programming.

eXtensible Markup Language XML

Becoming more common.

Installing RTedi

Login to system as the Road Tech Administrative user and check how up to date the Roadtech Administrative Toolkit is. Note the absolute minimum level for this to work is the set dated 18 February 2003.

whatever the date it is strongly recommended that you update to the latest release set.

Log out and log back in as the system administrative user (root). Invoke the Roadtech Administrative Toolkit RTadmin. Select SYSTEM utilities then System check menu followed by Check roadrunner links and ownerships

Exit back to the system prompt.

Configuring RTedi

I will look at the possibility of scripting this but for now. :-)
Please read this bit through careful before starting

If you do not already have one, create a shared group for use with EDI data links {datalink}, then name does not matter but you must use the same group throughout.

Create a user account, {edi2} again the name does not matter but what ever name you use you must be consistent.

Create this user with a primary group of {datalink}, and a group set including this group, and the group used for the COBOL programs {rm85}.

Make sure the Roadtech Administrative user {rr85} has {datalink} as part of its group set.

Log out and login as the new user {edi2}.

Start up RTedi in menu mode
RTedi -m

It will create a couple of directories and files, and then exit.

You now need to edit the example configuration file it has produced.
/home/{edi2}/bin/edi.var

The file is commented, it is recommended that you duplicate the example lines where you wish to change the default behaviour.

The default directory for files to be imported/exported is
/home/{edi2}/edi

Please do not try changing this to a users home directory, for security reasons the group, and group permissions must be different to those that are sensible on a users home directory.

If you restart RTedi in menu mode
RTedi -m

It should display a menu with a subset of the options displayed. check it all works.

FTP security

If your partner is going to use FTP to transfer files from their server to your main server despite the security problems. You can improve things a little with the following steps.

Edit /etc/passwd please make a backup copy first.

Find the line for the user created above {edi2}, duplicate this line.

Change the user name of the second copy, to {edi2ftp}.

Set the primary group for the new user name to nobody and the the group set to nobody,{datalink}.

Note even set as this any user who knows the password, for {edi2ftp} and as access to you partners server will be able to read a number of files on your server.

The command
find / -perm -004 ! -type l -ls
run as root will list files that are world readable, while the command
find / -perm -002 ! -type l -ls
will list the files that are world writable.

Alternatively run the following while logged in as {edi2ftp}


find / \( \( -group {nobody} -a -perm -040 \) -o -perm -400 \) ! -type l -ls

find / \( \( -group {datalink} -a -perm -040 \) -o -perm -400 \) ! -type l -ls

find / \( \( -group {nobody} -a -perm -020 \) -o -perm -200 \) ! -type l -ls

find / \( \( -group {datalink} -a -perm -020 \) -o -perm -200 \) ! -type l -ls

RTedi import methods

none

EDI_Import_Method=none

Must be the simplest, it disables import for this user.

program

EDI_Import_Method=program

This does very limited setup.

Current directory is set to ${EDI_Import_Dir} before invoking COBOL program. Environment is initialized to contain standard Roadtech Values, as well as RTedi's custom ones.

daemon

EDI_Import_Method=daemon

This method was added in 2012, for use with IBM WebSphere MQ. It does a very limited setup. It is expected that the program called will handle the data transfer its self.

Current directory is set to ${EDI_Import_Dir}. Environment is initialized to contain standard Roadtech Values, as well as RTedi's custom ones.

Script detaches from calling program and demonizes.

Daemon invokes COBOL program. Program alternates between active and sleeping states. It checks for the various system flags, if one of the relevant down flags is set the COBOL program exits returning control to the daemon.

As the COBOL program handles the data transfer, it is possible to have one program that does both import and export. In this case the control configuration will have the Export paths set, but EDI_Export_Method set to "none".

rredi

EDI_Import_Method=rredi

Runs the configured COBOL Program. Program should then collect any data available using the documented calls to communicate with the EDI daemon.

ftp

EDI_Import_Method=ftp

Checks that the user ____ does not have a current ftp connection to the system.

Looks in ${EDI_Import_Dir for file{s} matching ${EDI_Import_File}, or ${EDI_Import_Pattern} if set.

Files that appear to be closed will be copied to a history directory. If more than one was found (${EDI_Import_Pattern} set) they will be concatenated as ${EDI_Import_File} and then the configured program will be run.

Current directory is set to ${EDI_Import_Dir} before invoking COBOL program. Environment is initialized to contain standard Roadtech Values, as well as RTedi's custom ones.

Custom

EDI_Import_Method=custom

This allows for custom scripting. Within the directory ${HOME}/bin create a file called custom.lib.

within this file code a korn shell script edi_import_custom

If you wish to invoke a COBOL program, call the function edi_runcobol_import

RTedi export methods

none

EDI_Export_Method=none

Must be the simplest, it disable export for this configuration.

program

EDI_Export_Method=program

This does very limited setup.

Current directory is set to ${EDI_Import_Dir} before invoking COBOL program. Environment is initialized to contain standard Roadtech Values, as well as RTedi's custom ones.

daemon

EDI_Export_Method=daemon

This method was added in 2012, for use with IBM WebSphere MQ. It does a very limited setup. It is expected that the program called will handle the data transfer its self.

Current directory is set to ${EDI_Export_Dir}. Environment is initialized to contain standard Roadtech Values, as well as RTedi's custom ones.

Script detaches from calling program and demonizes.

Daemon invokes COBOL program. Program alternates between active and sleeping states. It checks for the various system flags, if one of the relevant down flags is set the COBOL program exits returning control to the daemon.

Custom

EDI_Export_Method=custom

This allows for custom scripting. Within the directory ${HOME}/bin create a file called custom.lib.

within this file code a korn shell script edi_export_custom

If you wish to invoke a COBOL program, call the function edi_runcobol_export

Using RTedi

RTedi can be used in a interactive menu mode, or in a programmed mode, either as part of another script, or from cron.

Menu Mode

Start RTedi -m from a shell. This will provide a menu screen with some basic configuration settings. The menu options allow various tasks to be performed.
14/02/2003      AIX Ver 4 rel 2
Default Print Queue := postscript

RMcobol Ver 6.61.00 Serial YR-0063-00000-0016
Data path := /usr/share/rr85

Import method :- rredi  Export method :- none
Company :- A User    :- EDI
Import Program W1EDIIN2

        1  Stop auto transfer.
        2  Start auto transfer.
           ----- ----- -----
        3  Display config.
           ----- ----- -----
        4  Import run.
        5  Export run.
        E  Exit.


                [ ]

Stop auto transfer

Sell explanatory, this option stops CRON from running transfer.

Start auto transfer

Configures cron entries for auto transfer.

The default configuration is to run every ten minutes with a 5 minute offset between the import program and the export program.

If the file $HOME/bin/cron exists this will be used rather than the built in timings.

Display config

Displays details on the current configuration.

Import run

Manually invokes the import method.

Export run

Manually invokes the export method.

cron

For import
0,10,20,30,40,50 * * * * RTedi -i
for export 5,15,25,35,45,55 * * * * RTedi -e

from a script

call RTedi with an option of -i or -e depending on whether you want to use the import or export method.

Customization

Customization takes the form of adding extra methods.

Notes for Programmers.

These notes relate to programs that are to be invoked under control of cron, and the RTedi scripts.

In addition to the RTedi, environment variables, all of the standard variables set when started from rr are also set.

Programs may leave an error log file EDIERR.ASC, in the current directory.

Import Programs

Either a third party program drops the data files for us, in a sub directory of the EDI users home directory, or we have to fetch them.

Fetching may be carried out by either these scripts or the called COBOL Program.

Export Programs

Either the COBOL Program writes the data directly to a network connection, or it writes 1 or more data files to a directory owned by the invoking EDI user. Usually this will be a sub directory of the EDI users home directory.

From here they can either be collected by a third party program, or pushed to their destination by the edi scripts.

COBOL programs that are called by other users may not, under any circumstances, write out data files for later transfer by an EDI export program or script. If a user action triggers the need to send an EDI message this trigger or event shall be written to a normal COBOL data file owned by the Roadrunner administrative user.

The program run by an EDI user to handle the EDI transfer will, read and delete the events from the event file, and write any necessary transfer file with the EDI user as owner. In this way the EDI script always has the necessary rights to clean up after a successful transfer.

Execution environment variables

RTedi loads the standard set of environment variables, for Roadtech scripts.

It also initializes the following values. These are either taken from the file $HOME/bin/edi.var, or initialized to default values.

These may all be used when writing custom import or export methods. Those that are exported may also be read by COBOL programs that use the GETENV call from libRTrm

rrcompanyExpARoadrunner company letter
EDI_UserNEDIRoadrunner user name
EDI_UidN132Roadrunner user id
EDI_CustomerN"Customer"Name for log entries and messages.
EDI_ErrorLevelN9Roadrunner Error level
EDI_DisplayFlagNNRoadrunner Display Flag
EDI_ParamiterModeN1Parameter mode
EDI_PlatformN$sysnamePlatform code (AIX, PC, SCO, CDOS)
EDI_GroupNdatalinkUnix group
EDI_wdirN$HOME/.rtediWorking directory for script logs.
 
EDI_Import_DirY$HOME/ediDirectory for import
EDI_Import_FileNedi.xmlFile name expected by COBOL program
EDI_Import_HostN"www.roadrunner.uk.com"For network methods
EDI_Import_MailToY${EDI_Import_MailTo:-$MailTo}Alerts
EDI_Import_MailCcY${EDI_Import_MailCc:-$MailCc}Alerts
EDI_Import_MethodYrrediMethod for import.
EDI_Import_PatternN$EDI_Import_FileImport File pattern
EDI_Import_ProgramNW1EDIIN2COBOL program
EDI_Import_URLN"/testxml/testjob.xml"For web methods
EDI_Import_FetchCmdN Command used by Fetch Methods
EDI_Import_SequenceN Pattern to extract a 3 digit sequence number from file name.
EDI_Import_TypeN Pattern to extract file type from file name.
EDI_Import_N  
 
EDI_Export_DirY$HOME/ediDirectory for export
EDI_Export_MethodYnoneMethod for export.
EDI_Export_ProgramNW1EDIOUTCOBOL program
EDI_Export_FileNedi.xml
EDI_Export_PatternN$EDI_Export_FileUser where the COBOL program drops more than one file.
EDI_Export_HostN"www.roadrunner.uk.com"For network methods
EDI_Export_URLN"/testxml/testjob.xml"For web methods
EDI_Export_MailToY${EDI_Export_MailTo:-$MailTo}Alerts
EDI_Export_MailCcY${EDI_Export_MailCc:-$MailCc}Alerts
EDI_Export_N  
EDI_Export_N  
EDI_Export_N  
EDI_N  
EDI_N  

EDI Program Execution

The configured program will be invoked as .
runcobol ${EDI_Import_Program} -a "$rrcompany ${EDI_User} ${EDI_Uid} ${EDI_ErrorLevel} ${EDI_DisplayFlag} ${EDI_ParamiterMode} ${EDI_Platform}" -k

To preserve compatibility it is required that any routines that are written, ignore additional fields at the end of the list. Additions must be Agreed by the programming department.

ParameterDefault ValueRange
$rrcompanyACompany letter [A-Z]
${EDI_User}EDIAny valid roadrunner user code
${EDI_Uid}132Numeric user ID used for temp files?
${EDI_ErrorLevel}9?
${EDI_DisplayFlag}N{Y|N}
${EDI_ParamiterMode1?
${EDI_Platform}$sysnameAIX, PC, SCO, CDOS, ...

Custom values for these parameters may be set the the edi.var file. Read from $HOME/bin

Custom Import/Export methods

Backups, This is important. RTedi checks for the existence of a backup flag, either global or DATAPATH specific, when invoked, before calling a method, and again before invoking runcobol.

As of 2012 it also checks for the EDI specific flags. These enable an administrator to temporarily suspend all EDIs or a specific EDI while, still running RoadRunner.

All of the built in methods will abort gracefully if a flag is found on any of these checks. No files shall under any circumstances, be altered within the file tree of any DATASET, unless the corresponding flags for the DATAPATH, have been checked within the preceding second.

Bookings Log

Please remember if you are creating or amending Bookings to write the appropriate entries to the Bookings Log file.

Error Messages

Any output to Standard Error via COBOL
DISPLAY UPON CONSOLE
statements will be logged, and regarded as indicating a failure of the program called.

Existing Import/Export Programs

I am not sure how accurate this list is, as I can find virtually no documentation.

If any of the programmers feel like owning up, and providing some documentation I will post links here.

ProgramEncodingTransportFormat
AAHIM1?File?
RRCABLQT?Fileeqcom?
RRCMAQT?FileCanada Maritime?
RRDRAYQTFixed WidthFileDraycote?
RREQCOQT?Fileeqcom?
RRMARIQT?File
RRMODM1??MOD?
RRMODM1??MOD?
RRPKSUPD?FileCurries?
RRRUNMAL?FTP
RRSENSCXMLTCPIPDirect via TcpOpen2Host
RRWHMQT??Self bill import and reconciliation
UTCALLRF??
W1EDIIN1?File
W1EDIIN2?File
W1EDIOUTFixed widthFile
W1PRDEDI?File
W2EDIIN1?File
W2EDIOT1?File
W2EDIOT2?File
W3IMPEDI?File
WHASCED1?FTP
WHEDIIN1Fixed width?
WHEDIIN2Fixed widthRTedid

ProgramEncodingTransportFormat
FFPROFQTFreight Forward?
RRARCQTCronable Archive run?
UTPATRFWCronable Diary Roll Forward?

ProgramEncodingTransportFormat
W3EDIIN1File
W3CSVIN1File
W3IMPED1File

Notes for Installs and Support.

Documentation for working configurations.