------------------------------------------------------------------------------ ReadMe for mfreq suite (c) 1994-2016 by Markus Reschke ------------------------------------------------------------------------------ Last edit: 2017-05-26 * About mfreq is a suite of tools for FTS style file requesting. It consists of a file index generator, filelist generator and a SRIF compatible file request handler. The original version of mfreq was developed for the venerable Commodore Amiga (tm) back in the 1990ies. Features: - case-sensitive and long filenames - filenames with spaces - magics - ifcico-style magic files - smart magics - password protected files - case-insensitive freqests (if enabled by cfg) - request limits - response netmail or text - dir.bbs and files.bbs - update of file description files - multiline support for filelists - tested with binkd and qico-xe * Updating If you update your current version of mfreq to a later one, always run mfreq-index to update your file indexes. There might be some changes in the file formats. * Copyright All files of the mfreq suite are copyrighted by Markus Reschke. * License The mfreq suite is distributed under the EUPL v1.1. For details see EUPL-v1.1.pdf or http://joinup.ec.europa.eu/software/page/eupl. ------------------------------------------------------------------------------ mfreq-index ------------------------------------------------------------------------------ * About This tool scans fileechos and directories to create a file index for a fast frequest processing later on. It also takes care of magic file names. * Usage The command line usage is: mfreq-index [-h/-?] [-c ] [-l ] -h/-? prints usage information (optional) -c configuration filepath (optional) -l log filepath (optional) Whithout the -c option the default filepath "/etc/fido/mfreq/index.cfg" will be used as configuration file. The -l option overrides the LogFile command of the configuration file. If neither the -l option nor the LogFile command are given, nothing will be logged. * Hints mfreq-index collects all file data in memory until it writes the data into an index file. If you have to deal with memory contraints please group your file areas and create an index file for each group. The frequest handler will accept multiple index files. Symbolic file links are not supported for file areas or magic files. They will be not followed. * Configuration The configuration file is more like a list of commands, which are processed by mfreq-index to create the file index. It's a simple ASCII text file. For a quick setup you'll find an example configuration "index.cfg" in the sample-cfg directory. + Basic Syntax Empty lines and lines starting with a "#" are treated as comments. Any other lines should contain a valid command. Commands mostly got additional arguments and options. Arguments with spaces, like a filepath, should be enclosed in quotation marks, for example "/fido/my files". The command parser also supports following quoted special characters: - \" for a literal " - \\ for a literal \ Outside quotations you may also use commas instead of whitespaces to separate options. + LogFile Command Syntax: LogFile The LogFile command specifies a filepath to be used as log file. If the -l option of the command line is used, it will overwrite the LogFile command. This command may be used only once. The best practice is to place LogFile as first command in the configuration. To prevent running several instances of mfreq-index writing the same file index at the same time the logfile will be locked. mfreq-index logs when it's started or ended. If you see a log entry that the program ended with errors, please run it manually and check the standard output for finding the cause of the error. + SetMode Command Syntax: SetMode [PathAliases] [BinarySearch] With SetMode you enable following features: Switch Feature ---------------------------------------------- PathAliases create path aliases BinarySearch enable binary filename search With PathAliases enabled mfreq-index creates automatically aliases for paths in the index data file and writes those aliases into the index alias file. That way the size of the index data file is decreased hugely. The drawback is that the request processing might take a little bit longer. This feature is supported just for regular files, not magics. If you got a large filebase with tons of files you should enable the binary search feature to speed up filename searching for simple requests (no wildcard as first character). Enable this feature for the request processor too. The speed gain is achieved by creating an additional index file and using the well known binary search algorithm. + Reset Command Syntax: Reset [SetMode] The Reset command does what you would expect :-) It resets several things back to their defaults: Keyword Action ----------------------------------------------------- SetMode resets all SetMode switches Excludes clears the list of file excludes + Magic Command Syntax: Magic File [PW ] The Magic command defines a frequest magic and its corresponding file. Add a magic command for each magic you want to serve. If you need to connect a single magic to several files add the same magic for each file. A password may be specified optionally to limit access. + SmartMagic Command Syntax: SmartMagic Path [Pattern ] [Latest] [PW ] This command allows you to create magics which select specific files within a directory based on given criteria. Following are supported: Keyword Selection --------------------------------------------------------------- Pattern files matching the name pattern Latest the latest file (modification time) You may combine the conditions and set a password optionally to limit access. A pattern supports following wildcards: - ? for matching a single character - * for matching any number of characters (including zero) Both wildcards may be used multiple times in a single pattern. + MagicPath Command Syntax: MagicPath The MagicPath command specifies a directory containing special magic files. Such a magic file is a simple text file containing filepaths (one filepath per line). The name of the magic file is used as name for the magic itself. This method is used by ifcico for example. You may use this command multiple times. If any file excludes are defined they'll be considered for the magic files, not for the filepaths in the magic files. + Exclude Command Syntax: Exclude To exclude specific files from being indexed use the Exclude command and specify a name or a name pattern. A pattern supports following wildcards: - ? for matching a single character - * for matching any number of characters (including zero) Both wildcards may be used multiple times in a single pattern. Each Exclude command will add the given name or pattern to a list, which is consulted when the MagicPath or AreaDir command is executed. + FileArea Command Syntax: FileArea [PW ] [Depth ] [AutoMagic] With the FileArea command the files inside the given directory will be added to the internal file index (stored in RAM). Use this command for each of your file areas/echos. If any file excludes are defined they'll be considered. A password may be specified optionally to limit access. The Depth argument enables directory recursion with a limit of the given levels. The level value should be in the range of 1 up to 10. The AutoMagic switch will create a magic for each file with the file's extension dropped off. For example: - myfile.zip -> myfile + SharedFileArea Command Syntax: SharedFileArea Path [PW ] [Depth ] [AutoMagic] [switches and options from mfreq-list] This is basically the same command like FileArea, but with a slightly changed syntax to share the command with mfreq-list. The mfreq-list specific switches and options are ignored. The idea is that you may put all your fileareas into an additional configuration file and share it by the Include command with mfreq-list. + Include Command Syntax: Include [Config ] If you like to spread your configuration over several files you may include those additional files with the Include command. Nesting is limited to 2 levels (main configuration is the first level). + Index Command Syntax: Index The Index command writes all files in the internal file index (stored in RAM) to the specified filepath. Following files will be written (based on SetMode): - .data - .lookup - .alias - .offset After writing the index the internal file index is emptied. Any new files are collected in the file index again, until another Index command will write those files to a new index and so on. This might come in handy if you got memory contraints or you need different file indexes for whatever reason. * ToDo / Feature Requests Some stuff which needs to be fixed or would be nice to have: - Verbose logging (including cfg switch)? ------------------------------------------------------------------------------ mfreq-list ------------------------------------------------------------------------------ * About The mfreq-list tool creates filelists and cleans up description files if requested. * Hints Symbolic file links are not supported for file areas . They will be not followed. * Description Files Brief overview of file description files and their syntax. All files may have lines ending with a CR (carriage return / 13). + simple files.bbs Plain ASCII file containing one or more lines for each file. Lines should not exceed a length of 78 characters. Format: [] [ ] Values: - name 8.3: DOS style filename (<8 chars name>.<3 chars extension>) - counter: download counter (usually []) + extended files.bbs Plain ASCII file containing one or more lines for each file. Lines should not exceed a length of 78 characters Format: [] [ ] Values: - name 8.3: DOS style filename (<8 chars name>.<3 chars extension>) - size: in bytes without commas - date: file date (usually MM-DD-YY) - counter: download counter (usually []) + dir.bbs Plain ASCII file containing one line with the description of the directory. + descript.ion Plain ASCII file containing one line for each file. Format: Long filenames are supported. Filenames with space(s) are enclosed in quotations marks. * Usage The command line usage is: mfreq-list [-h/-?] [-c ] [-l ] -h/-? prints usage information (optional) -c configuration filepath (optional) -l log filepath (optional) Whithout the -c option the default filepath "/etc/fido/mfreq/list.cfg" will be used as configuration file. The -l option overrides the LogFile command of the configuration file. If neither the -l option nor the LogFile command are given, nothing will be logged. * Configuration The configuration file is more like a list of commands, which are processed by mfreq-list to create filelists. It's a simple ASCII text file. For a quick setup you'll find an example configuration "list.cfg" in the sample-cfg directory. + Basic Syntax Empty lines and lines starting with a "#" are treated as comments. Any other lines should contain a valid command. Commands mostly got additional arguments and options. Arguments with spaces, like a filepath, should be enclosed in quotation marks, for example "/fido/my files". The command parser also supports following quoted special characters: - \" for a literal " - \\ for a literal \ Outside quotations you may also use commas instead of whitespaces to separate options. + LogFile Command Syntax: LogFile The LogFile command specifies a filepath to be used as log file. If the -l option of the command line is used, it will overwrite the LogFile command. This command may be used only once. The best practice is to place LogFile as first command in the configuration. To prevent running several instances of mfreq-list writing the same filelist(s) at the same time the logfile will be locked. mfreq-list logs when it's started or ended. If you see a log entry that the program ended with errors, please run it manually and check the standard output for finding the cause of the error. + Exclude Command Syntax: Exclude To exclude specific files from being listed use the Exclude command and specify a name or a name pattern. A pattern supports following wildcards: - ? for matching a single character - * for matching any number of characters (including zero) Both wildcards may be used multiple times in a single pattern. Each Exclude command will add the given name or pattern to a list, which is consulted when the FileArea command is executed. + Filelist Command Syntax: Filelist The Filelist command starts a new filelist and any list output is written to that file until a new filelist is defined or mfreq-list ends. If the given file already exists it will be overwritten. + AddText Command Syntax: AddText This command allows you to add some custom text to the current filelist. If the text includes any spaces enclose it with quotation marks, for example: AddText "This is my custom text." For an empty line use: AddText "" + Define files.bbs Command Syntax: Define files.bbs NameFormat [NameWidth ] [SizePos SizeFormat ] [DatePos DateFormat ] [CounterPos CounterFormat ] DescPos The command above defines the format of the files.bbs description file. It tells mfreq-list which data fields are used. Each call will reset all settings of the last call. The format definition is used for reading and writing. For all arguments the value is the position of the data field, starting with 1 for the first character (valid range: 1-120). A position of 40 means that the data field starts at character position 40. Arguments: * NameFormat Defines file name format. format description -------------------------------------------------- DOS 8.3 DOS style Long full name (you need to set NameWidth too!) * NameWidth Defines the maximum length of the filename. If is exceeded, the filename will be truncated with an asterisk ("*"). The valid range of is 12 up to 60. * SizePos SizeFormat Enables file size field. Defines start position and format. format description examples --------------------------------------------------------- Bytes-8 bytes without unit, max. 8 digits 1234 Unit-8 bytes with unit, max. 8 chars 12kB, 12KiB Short-8 bytes with short unit, max. chars 12k The size field is right-justified! So take care when defining the position! Leading zeros are omitted. When reading a files.bbs file mfreq-list accepts any valid format, not just the one you've set. * DatePos DateFormat Enables file date field. Defines start position and format. format description examples ----------------------------------- US MM-DD-YY 04-20-13 ISO YYYY-MM-DD 2013-04-20 * CounterPos CounterFormat Enables download counter field. Defines start position and format. format description examples ---------------------------------- Square-2 [nn] [12], [7] Square-3 [nnn] [345] Square-4 [nnnn] [6789] The counter field is right-justified! So take care when defining the position! Leading zeros are omitted. * DescPos Sets start position of file description. + Define filelist Command Syntax: Define filelist [NamePos ] NameFormat [NameWidth ] [SizePos SizeWidth SizeFormat ] [DatePos DateFormat ] [CounterPos CounterFormat ] DescPos DescWidth This command defines the output format of the filelist. It's similar to the "Define files.bbs" command. To support multiline output some data field positions may have a line number included using following syntax: - or . When omitting the line number, mfreq will assume the line number being 1. For example: - 1.10 means position 10 in line 1 - 1-10 same as above - 10 same as above - 3-40 means position 40 in line 3 The maximum line number supported is 5. To add empty lines skip line numbers. If you set NamePos to 2- an empty line is added before the file name. * [NamePos ] NameFormat [NameWidth ] Position (range: 1 - 120), width (range: 12 - 60) and format of file name: format description --------------------------------------------------- DOS 8.3 DOS style Long full name (you need to set NameWidth too!) * SizePos SizeWith SizeFormat Position (range: 1 - 120), width (range: 4 - 12) and format of file size: format description examples -------------------------------------------- Bytes bytes without unit 1234 Unit bytes with unit 12kB, 12KiB Short bytes with short unit 12k The size field is right-justified! So take care when defining the position! Leading zeros are omitted. * DatePos DateFormat Position (range: 1 - 120) and format of file date: format description examples ----------------------------------- US MM-DD-YY 04-20-13 ISO YYYY-MM-DD 2013-04-20 * CounterPos CounterFormat Position (range: 1 - 120) and format of download counter: format description examples ---------------------------------- Square-2 [nn] [12], [7] Square-3 [nnn] [345] Square-4 [nnnn] [6789] The counter field is right-justified! So take care when defining the position! Leading zeros are omitted. * DescPos DescWidth Position (range: 1 - 120) and width (range: 18-80) of description field. + InfoMode Command Syntax: InfoMode [dir.bbs] [files.bbs] [Update] [Strict] [Skips] [Relax] [SI-Units] Switch Feature ------------------------------------------------------- dir.bbs enable use of dir.bbs/DIR.BBS files.bbs enable use of files.bbs/FILES.BBS Update enable update of file description file Strict enable strict position checking for data fields Skips allow missing data fields behind filename Relax don't end processing of file description file on syntax errors AnyCase match file names case-insensitive SI-Units enable SI byte units IEC-Units enable IEC byte units output With InfoMode you set several switches for file description files and select sources for file descriptions. When you enable files.bbs, you have also to define it's format with the "Define files.bbs" command. In case the dir.bbs switch is set, mfreq-list reads the area description from that file if available. If you want mfreq-list to update the description file, e.g. removing deleted files, please set the Update switch. For strict position checking set the Strict switch, i.e. the start and end positions of the defined data fields must match exactly. The Skips switch allows missing data fields after the filename (premature end of line). With Relax set mfreq-list ignores syntax/format errors in file description files and keeps processing. So you'll get a filelist in any case. But be warned that the Relax switch may cause a lot of warnings being logged. If you set Relax and Update you can convert file descriptions files into the desired format. When you got issues with the case-sensitivity of filenames on disk and in the file description file, you can enable AnyCase to match the filenames case- insensitive. For example, if the filenames on disk are lower case and the filenames in the files.bbs are upper case. Make sure that you don't got any filename doubles when case-insensitive matching is applied. For double filenames on disk the first one found will get the file description. Any other double is treated as a new file not in the file description file yet. If you got doubles in the file description file, all descriptions of all doubles will be added to the first matching filename found on disk. The SI-Units switch enables metric unit prefixes for any input and output, i.e. 1 kB = 1000 Bytes. By default the binary prefix system is used. The IEC-Units switch enables output of the IEC prefix recommendations (1024 Bytes = 1 KiB) and overrides the SI-Units switch for output. + Reset Command Syntax: Reset [SetMode] As expected this command resets several things back to their defaults: Keyword Action ----------------------------------------------------- InfoMode resets all InfoMode switches Excludes clears the list of file excludes + Include Command Syntax: Include [Config ] If you like to spread your configuration over several files you may include those additional files with the Include command. Nesting is limited to 2 levels (main configuration is the first level). + FileArea Command Syntax: FileArea Path [Info ] The FileArea command scans a directory and writes a sorted list of files to the current filelist. Specify the file echos name and path. The Info argument overrides any other source of the fileareas description. + SharedFileArea Command Syntax: SharedFileArea Name Path [Info ] [switches and options from mfreq-index] This is basically the same command like FileArea, but with a slightly changed syntax to share the command with mfreq-index. The mfreq-index specific switches and options are ignored. The idea is that you may put all your fileareas into an additional configuration file and share it by the Include command with mfreq-index. * ToDo / Feature Requests Some stuff which needs to be fixed or would be nice to have: - Support special formatting of additional description lines in files.bbs? ("+ " / "| ") - Support comment lines in files.bbs? (" " / "--- ") - Support of descript.ion? - Support file_id.diz? Try to extract? - Ignore a missing directory and keep processing? - Verbose logging (including cfg switch)? ------------------------------------------------------------------------------ mfreq-srif ------------------------------------------------------------------------------ * About This is the SRIF frequest handler. It reads the mailers SRIF file, processes the file request while checking for any limits, writes a response list and creates a response netmail or textmail if configured. mfreq-srif requires an file index created by mfreq-index. * Usage The command line usage is: mfreq-srif [-h/-?] [-c ] [-l ] -s -h/-? prints usage information (optional) -c configuration filepath (optional) -l log filepath (optional) -s SRIF filepath (required) Whithout the -c option the default filepath "/etc/fido/mfreq/srif.cfg" will be used as configuration file. The -l option overrides the LogFile command of the configuration file. If neither the -l option nor the LogFile command are given, nothing will be logged. If you are running binkd you would add following to your binkd.cfg: exec "/path-to/mfreq-srif -s *S" *.req For qico-xe please add following to qico.conf: srifrp /path-to/mfreq-srif -s * Hints Symbolic file links are not supported for file areas or magic files. They will be not followed. All FTS addresses (AKAs) will be normalized into following format: :/.[@] The part is optional and nodes get point number 0. * Requests Requests with the following format are supported: [!] Each line should contain a single filename or pattern with an optional password separated by a space. Due to internal token processing the password may also be separated by comma or whitespaces. A filename or pattern with spaces needs to be enclosed in quotation marks, e.g. "my file.lha". Wildcards: - ? for matching a single character - * for matching any number of characters (including zero) Both wildcards may be used multiple times in a pattern. * NetMail/TextMail The frequest handler creates a response netmail or textmail for the requestor and adds the netmail packet or textmail file to the response list (with the delete-me-after-sent indicator). A result textblock is generated for each requested file/pattern: - [- ] The message is formatted like this: [MailHeader] [Result] ... [Result] [MailFooter] Password errors are treated like "no files found". By omitting any information about password problems the requestor is discouraged from password guessing. * Configuration It's a simple ASCII text file. For a quick setup you'll find an example configuration "srif.cfg" in the sample-cfg directory. + Basic Syntax Empty lines and lines starting with a "#" are treated as comments. Any other lines should contain a valid setting. Settings mostly got additional arguments and options. Arguments with spaces, like a filepath, should be enclosed in quotation marks, for example "/fido/my files". The command parser also supports following quoted special characters: - \" for a literal " - \\ for a literal \ Outside quotations you may also use commas instead of whitespaces to separate options. + LogFile Setting Syntax: LogFile The LogFile setting specifies a filepath to be used as log file. If the -l option of the command line is used, it will overwrite the LogFile command. This setting may be used only once. The best practice is to place LogFile as first setting in the configuration. mfreq-srif logs when it's started or ended. If you see a log entry that the program ended with errors, please run it manually within a test environment and check the standard output for finding the cause of the error. For the test environment simply create a pair of fake SRIF and request files. + SetMode Setting Syntax: SetMode [NetMail] [NetMail+] [TextMail] [RemoveReq] [AnyCase] [BinarySearch] [LogRequest] [SI-Units] [IEC-Units] With SetMode you enable following features: Switch Feature ---------------------------------------- NetMail send response netmail (type-2 packet, FTS-0001) NetMail+ send response netmail (type-2+ packet, FSC-0048) TextMail send response textmail (plain ASCII file) RemoveReq delete request file after processing AnyCase match file names case-insensitive BinarySearch enable binary filename search LogRequest log file requests in more detail SI-Units enable SI byte units IEC-Units enable IEC byte units output When netmail response is enabled, but the SRIF lacks the sysop name, the setting is automatically changed into textmail. Please set the RemoveReq switch if your mailer doesn't delete the request file after calling the frequest handler. If you've enabled the binary search feature for mfreq-index, enable it here too. This feature uses an additional index file to perform a binary search to speed up the processing of file requests. With LogRequest set mfreq-srif logs which files are requested and which are going to be sent. The SI-Units switch enables metric unit prefixes for any input and output, i.e. 1 kB = 1000 Bytes. By default the binary prefix system is used. The IEC-Units switch enables output of the IEC prefix recommendations (1024 Bytes = 1 KiB) and overrides the SI-Units switch for output. + Address Setting Syntax: Address :/[.][@] This defines your FTS addresses (AKAs). At least you should set your main AKA. Add an Address setting for each AKA. + MailDir setting Syntax: MailDir This setting is optional and specifies the path for response netmails or textmails. A good choice would be your outbound directory or /tmp. Used only once in the configuration. If not set, the default path "/var/tmp" will be used. Example: MailDir /tmp + MailHeader/MailFooter setting Syntax: MailHeader MailFooter Both settings allow you to add some custom text to the netmail or textmail response. MailHeader adds text at the top of the mail and MailFooter at the bottom. In between mfreq-srif will report any results of the request. The text is written to the mail in the same order as in the configuration. If the text includes any spaces enclose it with quotation marks, for example: MailHeader "This is my custom text." For an empty line use: MailHeader "" Some message readers won't display empty lines at the beginning of a netmail. To circumvent that just add a space, like: MailHeader " " MailHeader "bla bla" + Limit Setting Syntax: Limit [Files ] [Bytes ] [IfListed] This settings limits what a requester may retrieve. Actually it's a list of rules linked to a specfic FTS address pattern. The first matching rule will be used. The FTS address pattern is based on the normalized format (see hints) :/.[@] and supports following wildcards: - ? for matching a single character - * for matching any number of characters (including zero) Both wildcards may be used multiple times in a pattern. Some examples: pattern matches ----------------------------------------------------------- ?:* any system in a zone with one digit 2:* any system in zone 2 2:240/* any system in zone 2 and net 240 *@fidonet any system in domain fidonet * any system (for a rule of last resort) The Files argument limits the number of files which may be requested. And the Bytes argument defines the maximum number of bytes allowed. Both values should be in the range of 0 to 2147483000 (2^31 - 1). Zero means "none", i.e. that node won't get anything. There is also the special value "-1" for a unlimited number of files or bytes. For the byte value you may use following units: - kB for kilobytes (binary in default mode, decimal if SI units are enabled) - MB for megabytes (binary in default mode, decimal if SI units are enabled) - GB for gigabytes (binary in default mode, decimal if SI units are enabled) - KiB for kibibytes - MiB for mebibytes - GiB for gibibytes Place the unit directly behind the value or enclose the term in quotation marks if you insert a space: - 10MB - "10 MB" Some examples: Limit 1:2/3.4 Files 0 -> deny any files for 1:2/3.4 Limit 2:240/* Files 10 Bytes -1 -> net 2:240 get's up to 10 files with unlimited size Limit * Files 10 Bytes 5MiB -> anyone else get's up to 10 files with a total size of 5MB With the IfListed switch set, the rule is only used if the node is listed in a nodelist. That information is provided by the mailer via the SRIF file. If you define any rules, it's strongly recommended to define also a rule of last resort, like Limit * Files 10 Bytes 2MB The following defaults are used if no limit is given at all: - 20 files - 2000000 bytes + Index Setting Syntax: Index [IfMounted ] The Index setting adds a file index to be used for file searching. You may add several file indexes if you like. The argument will be expanded to match all the data files for a single index: - .data - .lookup - .alias - .offset For removable media you can set the IfMounted option. In that case the index is only considered if a filesystem is mounted at the given path. Of course you should have created a dedicated index with mfreq-index before. * ToDo / Feature Requests Some stuff which needs to be fixed or would be nice to have: - Support Time setting in SRIF file? - Exit if a maximum runtime is exceeded? ------------------------------------------------------------------------------ misc stuff ------------------------------------------------------------------------------ * Log Files Format of entries in the log file:
:: [] * Latest Version The latest version is available via: - frequest from 2:240/1661 (magic MFREQ) - GIT at https://github.com/ftnapps/mfreq - GIT at sourceforge.net/p/ftnapps/mfreq (outdated) * Forks & Other OSs Lars maintains a modfied version for OS X at https://github.com/larsks/mfreq. * Contact the Author EMail: madires@theca-tabellaria.de Netmail: Markus Reschke 2:240/1661 * Credits A big thank you to: Paul Quinn, 3:640/384, for beta-testing and some great feature suggestions. Rj Clay, 1:120/419, for the GIT repository at sourceforge (FTN Applications) and Github (ftnapps). Lars Kellogg-Stedman, 1:322/761, for the OS X version. ------------------------------------ EOF -------------------------------------