cacache
cacache
is a Node.js library for managing local key and content address caches. It's really fast, really good at concurrency, and it will never give you corrupted data, even if cache files get corrupted or manipulated.
On systems that support user and group settings on files, cacache will match the uid
and gid
values to the folder where the cache lives, even when running as root
.
It was written to be used as npm's local cache, but can just as easily be used on its own.
Translations: español
Install
$ npm install --save cacache
Table of Contents
Example
Features
Extraction by key or by content address (shasum, etc)
Subresource Integrity web standard support
Multi-hash support - safely host sha1, sha512, etc, in a single cache
Automatic content deduplication
Fault tolerance (immune to corruption, partial writes, process races, etc)
Consistency guarantees on read and write (full data verification)
Lockless, high-concurrency cache access
Streaming support
Promise support
Pretty darn fast -- sub-millisecond reads and writes including verification
Arbitrary metadata storage
Garbage collection and additional offline verification
Thorough test coverage
There's probably a bloom filter in there somewhere. Those are cool, right? 🤔
Contributing
The cacache team enthusiastically welcomes contributions and project participation! There's a bunch of things you can do if you want to contribute! The Contributor Guide has all the information you need for everything from reporting bugs to contributing entire new features. Please don't hesitate to jump in if you'd like to, or even ask us questions if something isn't clear.
All participants and maintainers in this project are expected to follow Code of Conduct, and just generally be excellent to each other.
Please refer to the Changelog for project history details, too.
Happy hacking!
API
> cacache.ls(cache) -> Promise<Object>
> cacache.ls(cache) -> Promise<Object>
Lists info for all entries currently in the cache as a single large object. Each entry in the object will be keyed by the unique index key, with corresponding get.info
objects as the values.
Example
> cacache.ls.stream(cache) -> Readable
> cacache.ls.stream(cache) -> Readable
Lists info for all entries currently in the cache as a single large object.
This works just like ls
, except get.info
entries are returned as 'data'
events on the returned stream.
Example
> cacache.get(cache, key, [opts]) -> Promise({data, metadata, integrity})
> cacache.get(cache, key, [opts]) -> Promise({data, metadata, integrity})
Returns an object with the cached data, digest, and metadata identified by key
. The data
property of this object will be a Buffer
instance that presumably holds some data that means something to you. I'm sure you know what to do with it! cacache just won't care.
integrity
is a Subresource Integrity string. That is, a string that can be used to verify data
, which looks like <hash-algorithm>-<base64-integrity-hash>
.
If there is no content identified by key
, or if the locally-stored data does not pass the validity checksum, the promise will be rejected.
A sub-function, get.byDigest
may be used for identical behavior, except lookup will happen by integrity hash, bypassing the index entirely. This version of the function only returns data
itself, without any wrapper.
Note
This function loads the entire cache entry into memory before returning it. If you're dealing with Very Large data, consider using get.stream
instead.
Example
> cacache.get.stream(cache, key, [opts]) -> Readable
> cacache.get.stream(cache, key, [opts]) -> Readable
Returns a Readable Stream of the cached data identified by key
.
If there is no content identified by key
, or if the locally-stored data does not pass the validity checksum, an error will be emitted.
metadata
and integrity
events will be emitted before the stream closes, if you need to collect that extra data about the cached entry.
A sub-function, get.stream.byDigest
may be used for identical behavior, except lookup will happen by integrity hash, bypassing the index entirely. This version does not emit the metadata
and integrity
events at all.
Example
> cacache.get.info(cache, key) -> Promise
> cacache.get.info(cache, key) -> Promise
Looks up key
in the cache index, returning information about the entry if one exists.
Fields
key
- Key the entry was looked up under. Matches thekey
argument.integrity
- Subresource Integrity hash for the content this entry refers to.path
- Filesystem path where content is stored, joined withcache
argument.time
- Timestamp the entry was first added on.metadata
- User-assigned metadata associated with the entry/content.
Example
> cacache.get.hasContent(cache, integrity) -> Promise
> cacache.get.hasContent(cache, integrity) -> Promise
Looks up a Subresource Integrity hash in the cache. If content exists for this integrity
, it will return an object, with the specific single integrity hash that was found in sri
key, and the size of the found content as size
. If no content exists for this integrity, it will return false
.
Example
> cacache.put(cache, key, data, [opts]) -> Promise
> cacache.put(cache, key, data, [opts]) -> Promise
Inserts data passed to it into the cache. The returned Promise resolves with a digest (generated according to opts.algorithms
) after the cache entry has been successfully written.
Example
> cacache.put.stream(cache, key, [opts]) -> Writable
> cacache.put.stream(cache, key, [opts]) -> Writable
Returns a Writable Stream that inserts data written to it into the cache. Emits an integrity
event with the digest of written contents when it succeeds.
Example
> cacache.put options
> cacache.put options
cacache.put
functions have a number of options in common.
opts.metadata
Arbitrary metadata to be attached to the inserted key.
opts.size
If provided, the data stream will be verified to check that enough data was passed through. If there's more or less data than expected, insertion will fail with an EBADSIZE
error.
opts.integrity
If present, the pre-calculated digest for the inserted content. If this option if provided and does not match the post-insertion digest, insertion will fail with an EINTEGRITY
error.
algorithms
has no effect if this option is present.
opts.algorithms
Default: ['sha512']
Hashing algorithms to use when calculating the subresource integrity digest for inserted data. Can use any algorithm listed in crypto.getHashes()
or 'omakase'
/'お任せします'
to pick a random hash algorithm on each insertion. You may also use any anagram of 'modnar'
to use this feature.
Currently only supports one algorithm at a time (i.e., an array length of exactly 1
). Has no effect if opts.integrity
is present.
opts.memoize
Default: null
If provided, cacache will memoize the given cache insertion in memory, bypassing any filesystem checks for that key or digest in future cache fetches. Nothing will be written to the in-memory cache unless this option is explicitly truthy.
If opts.memoize
is an object or a Map
-like (that is, an object with get
and set
methods), it will be written to instead of the global memoization cache.
Reading from disk data can be forced by explicitly passing memoize: false
to the reader functions, but their default will be to read from memory.
> cacache.rm.all(cache) -> Promise
> cacache.rm.all(cache) -> Promise
Clears the entire cache. Mainly by blowing away the cache directory itself.
Example
> cacache.rm.entry(cache, key) -> Promise
> cacache.rm.entry(cache, key) -> Promise
Alias: cacache.rm
Removes the index entry for key
. Content will still be accessible if requested directly by content address (get.stream.byDigest
).
To remove the content itself (which might still be used by other entries), use rm.content
. Or, to safely vacuum any unused content, use verify
.
Example
> cacache.rm.content(cache, integrity) -> Promise
> cacache.rm.content(cache, integrity) -> Promise
Removes the content identified by integrity
. Any index entries referring to it will not be usable again until the content is re-added to the cache with an identical digest.
Example
> cacache.clearMemoized()
> cacache.clearMemoized()
Completely resets the in-memory entry cache.
> tmp.mkdir(cache, opts) -> Promise<Path>
> tmp.mkdir(cache, opts) -> Promise<Path>
Returns a unique temporary directory inside the cache's tmp
dir. This directory will use the same safe user assignment that all the other stuff use.
Once the directory is made, it's the user's responsibility that all files within are given the appropriate gid
/uid
ownership settings to match the rest of the cache. If not, you can ask cacache to do it for you by calling tmp.fix()
, which will fix all tmp directory permissions.
If you want automatic cleanup of this directory, use tmp.withTmp()
Example
> tmp.fix(cache) -> Promise
> tmp.fix(cache) -> Promise
Sets the uid
and gid
properties on all files and folders within the tmp folder to match the rest of the cache.
Use this after manually writing files into tmp.mkdir
or tmp.withTmp
.
Example
> tmp.withTmp(cache, opts, cb) -> Promise
> tmp.withTmp(cache, opts, cb) -> Promise
Creates a temporary directory with tmp.mkdir()
and calls cb
with it. The created temporary directory will be removed when the return value of cb()
resolves, the tmp directory will be automatically deleted once that promise completes.
The same caveats apply when it comes to managing permissions for the tmp dir's contents.
Example
Subresource Integrity Digests
For content verification and addressing, cacache uses strings following the Subresource Integrity spec. That is, any time cacache expects an integrity
argument or option, it should be in the format <hashAlgorithm>-<base64-hash>
.
One deviation from the current spec is that cacache will support any hash algorithms supported by the underlying Node.js process. You can use crypto.getHashes()
to see which ones you can use.
Generating Digests Yourself
If you have an existing content shasum, they are generally formatted as a hexadecimal string (that is, a sha1 would look like: 5f5513f8822fdbe5145af33b64d8d970dcf95c6e
). In order to be compatible with cacache, you'll need to convert this to an equivalent subresource integrity string. For this example, the corresponding hash would be: sha1-X1UT+IIv2+UUWvM7ZNjZcNz5XG4=
.
If you want to generate an integrity string yourself for existing data, you can use something like this:
You can also use ssri
to have a richer set of functionality around SRI strings, including generation, parsing, and translating from existing hex-formatted strings.
> cacache.verify(cache, opts) -> Promise
> cacache.verify(cache, opts) -> Promise
Checks out and fixes up your cache:
Cleans up corrupted or invalid index entries.
Custom entry filtering options.
Garbage collects any content entries not referenced by the index.
Checks integrity for all content entries and removes invalid content.
Fixes cache ownership.
Removes the
tmp
directory in the cache and all its contents.
When it's done, it'll return an object with various stats about the verification process, including amount of storage reclaimed, number of valid entries, number of entries removed, etc.
Options
opts.filter
- receives a formatted entry. Return false to remove it. Note: might be called more than once on the same entry.
Example
> cacache.verify.lastRun(cache) -> Promise
> cacache.verify.lastRun(cache) -> Promise
Returns a Date
representing the last time cacache.verify
was run on cache
.
Example
Last updated