FILE-FINDER

Warning: This library is currently experimental. While perfectly usable as is, the application programming interface is prone to change in the future.

<2024-08-30>: file-finder is on Ultralisp and is included in CIEL.

<2024-10-16>: file-finder is on Quicklisp, since the 2024-10 release.

Please report any issue or suggestion, including:

• Better function, variable, slot, class or package naming.
• Better function arguments.
• Filesystem issues.

Features

Enable rapid file search, inspection and manipulation.

• A file class which embeds the path and metadata such as size, ctime/mtime/atime.
• Various path manipulation functions which supersede Common Lisp pathname related functions.

  Using file instead of pathname saves us from many pitfalls, for instance path with wildcards (such as `[`, `*`) are no longer special.

• finder and finder* which return a list of files matching predicates.

  finder is a convenience wrapper around finder*. The latter has more options.

• A file-finder:syntax readtable to enable the #f"/path/to/file" syntax, which mimicks #p for pathnames.

In practice, it mostly supersedes:

• Common Lisp pathnames (at least for existing files).
• find for recursive and programmable file search. Unlike find, finder’s predicates are extensible.
• du
• touch

Note that FILE-FINDER is not meant to manipulate arbitrary paths of non-existing files. Consider using ppath instead.

Portability

For now this is only tested on Unix-based systems. Help welcome if you need support for another system.

File search and recursive listing

List all files in the current directory, recursively:

(finder)
; => (#F"~/co…/file-finder/LICENSE"
;     #F"~/co…/file-finder/ffprobe.lisp"
;     #F"~/co…/file-finder/file.lisp"
;     #F"~/co…/file-finder/file-finder.asd"
;     #F"~/co…/file-finder/mediafile.lisp"
;     ...)

;; Same, with a given root, without descending into hidden directories and
;; without descending more than one level:
(finder* :root (file ".") :recur-predicates (list (complement #'hidden?)
                                                  (depth< 2)))

List files matching all the given predicates:

(finder "fil" (extension= "lisp"))
; => (#F"~/co…/file-finder/file.lisp")
;; Passing a string as a predicate specifier is equivalent to `path~':
(finder (path~ "fil") (extension= "lisp"))

;; Passing a pathname is equivalent to `path$' (match end of path).

We include useful predicate or predicate generators you can complete against, see the list below.

Passing a list of predicate specifiers connects them with a logical or. In other words, it returns the files matching at least one of the predicate specifiers:

(finder (list "fil" (extension= "asd")))
;; this returns many results:
; => (#F"~/co…/file-finder/file.lisp" #F"~/co…/file-finder/file-finder.asd" #F"~/co…/file-finder/mediafile.lisp")

;; While this finds 1 file:
(finder "fil" (extension= "asd"))

To get the file names, use path:

(mapcar #'path *)
; => "~/co…/file-finder/mediafile.lisp"  (sans the #F reader macro)

For more complex predicate list nesting, you can leverage alexandria:disjoin and alexandria:conjoin.

Hidden files and directories

file-finder doesn’t return hidden files by default (on Linux, files starting with “.”) and doesn’t visit hidden directories.

You can search in them by setting 2 parameters:

;; List hidden files, descend hidden directories.
(let ((*include-hidden-files* t)
      (*descend-hidden-directories* t))
  (finder))

Exclude directories

The node_modules directory is excluded by default.

You can enlarge the list into *exclude-directories* (use pathnames with #p"directory/" instead of strings).

List of predicates

Most predicates are functions that accept one or many strings as arguments. In that case, they return a lambda function, that receives the file object as argument. For example:

(finder (lambda (file-object)
   (str:containsp "lisp" (path file-object)))) ;; get the path name from the file object.

It is possible to use predicates that don’t take arguments.

In predicates.lisp, see:

• path~: matches when one of the path elements is contained in the file path.
  • every-path~: same checks on the file path, but uses a logical and.
• path$: matches when one of the path suffixes matches the file path.
• name=: matches when one of the names matches the file name (case sensitive).
• iname=: matches when one of the names matches the file name (case insensitive).
• name~: matches when one of the names is contained in the file basename (and not the whole path), case sensitive.
  • every-name~: same checks on the file basename, but uses a logical and.
• iname~: matches when one of the names is contained in the file, case insensitive.
• depth<: matches when the argument file is in a subdirectory of ROOT less deep than LEVEL.
• elf-binary? and elf-library?.

Make inspectable file objects

(file "file-finder.asd")
; => #F"~/co…/file-finder/file-finder.asd"

(inspect *)
; =>
The object is a STANDARD-OBJECT of type FILE-FINDER/FILE::FILE.
0. PATH: "/home/lisp-maintainers/projects/file-finder/file-finder.asd"
1. INODE: 3223494
2. LINK-COUNT: 1
4. SIZE: 1565
5. DISK-USAGE: 12288
8. CREATION-DATE: @2023-11-16T19:08:16.000000+01:00
9. MODIFICATION-DATE: @2023-11-16T19:08:16.000000+01:00
10. ACCESS-DATE: @2024-04-22T17:50:58.000000+02:00

;; Enable reader macro:
(named-readtables:in-readtable file-finder:syntax)
; => #<NAMED-READTABLE READTABLE {1003035363}>

;; Now you can use the #f syntax:
#f"file-finder.asd"
; => #F"~/co…/file-finder/file-finder.asd"

;; Recursive disk-usage, in bytes.
(disk-usage #f".")
; => 1298432

;; Custom printer with abbreviations disabled:
(setf *print-abbreviation-threshold* 0
      *print-abbreviate-home?* nil
      *print-size?* t
      *print-date?* t)
; => #F"/home/ambrevar/common-lisp/file-finder/file-finder.asd 348 Feb 28 16:56"

Familiar path manipulation functions

(separator)
; => "/"

(current-directory)
; => #F"~/co…/file-finder/"

(extension #f"file-finder.asd")
; => "asd"
(basename #f"../file-finder/file-finder.asd")
; => "file-finder.asd"
(parent #f"file-finder.asd")
; => #F"~/co…/file-finder/"
(relative-path #f"file-finder.asd" #f"..")
; => "file-finder/file-finder.asd

(file? #f"file-finder.asd")
; => T
(directory? #f"file-finder.asd")
; => NIL
(let ((f #f"file-finder.asd"))
  (delete-file f)
  (exists? f))
; => NIL

Changelog and acknowledgements

This library was cloned from fof (file-object finder) by @ambrevar. FOF is richer in that its file object also gives: user and group IDs, stats, file kind (regular, executable) and permissions in user-readable format (:user-read, :user-write etc).

We are most interested in the search features, hence the clone and the cleanup. We could re-include some of the removed features by relying on the newer file-attributes library. PR welcome.

We did the following changes in this fork.

2024-04

Removed Osicat dependency and related features

• removed dependency on osicat, and added dependency on file-attributes
• removed: getting the user, group, stats, permissions, of a file and the related finder predicates (user, group, kind, executable).

In fof, an object has these slots:

0. PATH: "/home/lisp-maintainers/projects/file-finder/file-finder.asd"
1. INODE: 3223494
2. LINK-COUNT: 1
3. KIND: :REGULAR-FILE
4. SIZE: 1565
5. DISK-USAGE: 12288
6. USER-ID: 1000
7. GROUP-ID: 1000
8. CREATION-DATE: @2023-11-16T19:08:16.000000+01:00
9. MODIFICATION-DATE: @2023-11-16T19:08:16.000000+01:00
10. ACCESS-DATE: @2024-04-22T17:50:58.000000+02:00
11. PERMISSIONS: (:USER-READ :USER-WRITE :GROUP-READ :GROUP-WRITE :OTHER-READ)

In file-finder, these ones:

0. PATH: "/home/vince/quicklisp/local-projects/fof/readme.org"
1. INODE: 0
2. LINK-COUNT: 0
3. KIND: :REGULAR-FILE
4. SIZE: 0
5. DISK-USAGE: 0
6. CREATION-DATE: @1970-01-01T01:00:00.000000+01:00     <--- 1970 here
7. MODIFICATION-DATE: @2024-04-24T13:45:38.000000+02:00
8. ACCESS-DATE: @2024-04-24T13:45:38.000000+02:00

• removed dependency on hu.dwim.defclass-star.
• removed dependency on trivia.
• changed from package-inferred systems to traditional system (list dependencies in the .asd)
• changed from one package per file to one package for the project: simplify symbols mangling, no need to import&reexport and a bit easier to type for the user.
• added: don’t return hidden files by default and don’t recur into hidden directories (typically, .git/) (merged unmerged MR #2 of FOF).
• added: exclude node_modules/ directory by default.

Removed libmagic/magicffi dependency and dropped support for mime-type and file encoding

• magicffi is a FFI wrapper around libmagic, a unix command to recognize the type of data contained in a file. It makes installation and distribution less straightforward, and is unix only. FOF used it to get the file mime type and file encoding, and to provide related filters.
  • see also trivial-mimes. It first does a simple association file type -> mime type, without looking at the file content. If that fails, it consults the file unix command.