Python: module datazen.classes.file_info

datazen.classes.file_info_cache

index
/home/vkottler/src/vkottler/workspace/datazen/datazen/classes/file_info_cache.py

datazen - A class for storing metadata about files that have been loaded.

Modules

logging

shutil

time

Classes

builtins.object

FileInfoCache

class FileInfoCache(builtins.object)

FileInfoCache(cache_dir: str = None, logger: logging.Logger = <Logger datazen.classes.file_info_cache (WARNING)>) -> None

Provides storage for file hashes and lists that have been loaded.

Methods defined here:

__init__(self, cache_dir: str = None, logger: logging.Logger = <Logger datazen.classes.file_info_cache (WARNING)>) -> None: Construct an empty cache or optionally load from a directory.

check_hit(self, sub_dir: str, path: str, also_cache: bool = True) -> bool: Determine if a given file already exists with its current hash in the
cache, if not return False and optionally add it to the cache.

clean(self) -> None: Remove cached data from the file-system.

describe(self) -> None: Describe this cache's contents for debugging purposes.

get_data(self, name: str) -> datazen.load.LoadedFiles: Get the tuple version of cached data.

get_hashes(self, sub_dir: str) -> Dict[str, Any]: Get the cached, dictionary of file hashes for a certain key.

get_loaded(self, sub_dir: str) -> List[str]: Get the cached, list of loaded files for a certain key.

load(self, cache_dir: str) -> None: Load data from a directory.

write(self, out_type: str = 'json') -> None: Commit cached data to the file-system.

Data descriptors defined here:

__dict__: dictionary for instance variables

__weakref__: list of weak references to the object

Functions

cmp_loaded_count(cache_a: datazen.classes.file_info_cache.FileInfoCache, cache_b: datazen.classes.file_info_cache.FileInfoCache, name: str) -> int: Compute the total difference in file counts (for a named group)
between two caches.

cmp_loaded_count_from_set(cache_a: datazen.classes.file_info_cache.FileInfoCache, cache_b: datazen.classes.file_info_cache.FileInfoCache, name: str, files: List[str]) -> int: Count the number of files uniquely loaded to one cache but not the other.

cmp_total_loaded(cache_a: datazen.classes.file_info_cache.FileInfoCache, cache_b: datazen.classes.file_info_cache.FileInfoCache, known_types: List[str], load_checks: Dict[str, List[str]] = None, logger: logging.Logger = <Logger datazen.classes.file_info_cache (WARNING)>) -> int: Compute the total difference in file counts for a provided set of named
groups.

copy(cache: datazen.classes.file_info_cache.FileInfoCache) -> datazen.classes.file_info_cache.FileInfoCache: Copy one cache into a new one.

meld(cache_a: datazen.classes.file_info_cache.FileInfoCache, cache_b: datazen.classes.file_info_cache.FileInfoCache) -> None: Promote all updates from cache_b into cache_a.

remove_missing_hashed_files(data: Dict[str, Any], removed_data: Dict[str, List[str]]) -> Dict[str, Any]: Assign new hash data based on the files that are still present.

remove_missing_loaded_files(data: Dict[str, Any]) -> Dict[str, Any]: Audit list elements in a dictionary recursively, assume the data is String
and the elements are filenames, assign a new list for all of the elements
that can be located.

sync_cache_data(cache_data: Dict[str, Any], removed_data: Dict[str, List[str]]) -> Dict[str, Any]: Before writing a cache to disk we want to de-duplicate items in the loaded
list and remove hash data for files that were removed so that if they
come back at the same hash, it's not considered already loaded.

time_str(time_s: float) -> str: Concert a timestamp to a String.

Data
		DATA_DEFAULT = {'hashes': defaultdict(<function <lambda> at 0x7f2d3da03ba0>, {}), 'loaded': defaultdict(<class 'list'>, {}), 'meta': {'version': '3.1.4'}} Dict = typing.Dict GenericStrDict = typing.Dict[str, typing.Any] LOG = <Logger datazen.classes.file_info_cache (WARNING)> List = typing.List VERSION = '3.1.4'