squel/steam-manifest

Fork 0

feat: handle multiple input manifests #14

Merged

squel merged 13 commits from squel/failure into main

2025-01-26 15:49:41 +00:00

AGit

squel commented

2024-12-23 23:56:23 +00:00

Owner

moving the failure checking to func main allows us
to handle multiple manifest inputs as follows:

if there are files that have been updated
(changed) as specified in a more recent
manifest-file and thus in a later iteration, we
will not show errors because they are checking
against an old hash as specified in an old
manifest file
if there are files missing in the current
iteration, but present in a later manifest-file,
we will not show errors because they are missing

this is with the following caveats:

it is assumed that the last input is the most
recent or otherwise "most correct". this will
be an issue because the ManifestID by
Valve/Steam (secondary part of the filename:
AppID_ManifestID.manifest) is not
incrementative and thus not in alphabetical
order (which is what shells will expand
wildcards to). an improvement will be check the
manifests' CreationTime metadata
failed (hash mismatch or missing) entries will
be shown last in the output, instead of in the
order in which they completed processing. this
is fine to me however, since it was
multithreaded and thus semi-random to begin with
and does not impact UX whatsover. if anything
this is an improvement imo
code structure wise it is kinda bad to have such
a long Action, plus the failures are counted in
two different places but ¯_(ツ)_/¯

moving the failure checking to func main allows us to handle multiple manifest inputs as follows: - if there are files that have been updated (changed) as specified in a more recent manifest-file and thus in a later iteration, we will not show errors because they are checking against an old hash as specified in an old manifest file - if there are files missing in the current iteration, but present in a later manifest-file, we will not show errors because they are missing this is with the following caveats: - it is assumed that the last input is the most recent or otherwise "most correct". this **will be** an issue because the ManifestID by Valve/Steam (secondary part of the filename: AppID_ManifestID.manifest) is **not** incrementative and thus not in alphabetical order (which is what shells will expand wildcards to). an improvement will be check the manifests' CreationTime metadata - failed (hash mismatch or missing) entries will be shown last in the output, instead of in the order in which they completed processing. this is fine to me however, since it was multithreaded and thus semi-random to begin with and does not impact UX whatsover. if anything this is an improvement imo - code structure wise it is kinda bad to have such a long Action, plus the failures are counted in two different places but ¯\_(ツ)_/¯

squel added 4 commits

2024-12-23 23:56:23 +00:00

sha1sum d9fb63d413

multithread sha1sum 1942a9c1b8

additional arg 7a0b2f0aab

missing files 82c2f2cf62

squel commented

2024-12-23 23:58:31 +00:00

Author

Owner

This is handling failure checking as outlined in:

 	// TODO: ideally, this would be handled/checked at the cCtx.Args() for-loop,
 	// so incase there are multiple manifest files input by the user (multipart
 	// manifest), it will not trigger a warning/failure state if entries are
 	// declared or updated in another manifest-file. so this will be the case
 	// for both failure states: incorrect hash and missing file

This is handling failure checking as outlined in: https://git.squel.xyz/squel/steam-manifest/src/commit/82c2f2cf62d8895a8ccacb5f7f7182de93a2ef1c/utils/check.go#L17-L21

squel commented

2024-12-24 00:01:08 +00:00

Author

Owner

EDIT: 78dafae48a

~~Todo: handle CreationTime metadata, instead of assuming the last one to be correct, as outlined in:~~

squel/steam-manifest@6811194928/main.go (L105-L109)

~~and 68111949285b158b86ec620d8eae90770d38a550 's commit message:~~

this is with the following caveats:

it is assumed that the last input is the most
recent or otherwise "most correct". this will
be an issue because the ManifestID by
Valve/Steam (secondary part of the filename:
AppID_ManifestID.manifest) is not
incrementative and thus not in alphabetical
order (which is what shells will expand
wildcards to). an improvement will be check the
manifests' CreationTime metadata
(...)

EDIT: 78dafae48aa8ab82701a26aed3e8abf425fb95ff ~~Todo: handle CreationTime metadata, instead of assuming the last one to be correct, as outlined in:~~ https://git.squel.xyz/squel/steam-manifest/src/commit/68111949285b158b86ec620d8eae90770d38a550/main.go#L105-L109 ~~and 68111949285b158b86ec620d8eae90770d38a550 's commit message:~~ > this is with the following caveats: > - it is assumed that the last input is the most > recent or otherwise "most correct". this **will > be** an issue because the ManifestID by > Valve/Steam (secondary part of the filename: > AppID_ManifestID.manifest) is **not** > incrementative and thus not in alphabetical > order (which is what shells will expand > wildcards to). an improvement will be check the > manifests' CreationTime metadata > (...)

squel changed title from ~~handle multiple input manifests~~ to failure checking: handle multiple input manifests

2024-12-24 00:01:22 +00:00

squel force-pushed squel/failure from 6811194928 to 5741bf6f80

2024-12-24 00:18:52 +00:00

Compare

squel commented

2024-12-24 00:23:09 +00:00

Author

Owner

5741bf6f80 again to resolve conflicts... manually reapplying the changes made because rebase was being a PITA..

differences being:

replace [20]byte with [sha1.Size]byte
remove --show-failed argument to show missing entries by default (argument was intended to be called --show-missing)

5741bf6f8098c78a30647f7b57b6ef6a3635756d again to resolve conflicts... manually reapplying the changes made because rebase was being a PITA.. differences being: - replace [`[20]byte`](https://git.squel.xyz/squel/steam-manifest/src/commit/68111949285b158b86ec620d8eae90770d38a550/main.go#L96) with [`[sha1.Size]byte`](https://git.squel.xyz/squel/steam-manifest/src/commit/5741bf6f8098c78a30647f7b57b6ef6a3635756d/main.go#L92) - remove [`--show-failed`](https://git.squel.xyz/squel/steam-manifest/src/commit/68111949285b158b86ec620d8eae90770d38a550/main.go#L89-L94) [argument](https://git.squel.xyz/squel/steam-manifest/src/commit/68111949285b158b86ec620d8eae90770d38a550/main.go#L157) to show missing entries by default (argument was intended to be called `--show-missing`)

squel added 3 commits

2024-12-25 21:09:30 +00:00

sort based on CreationDate metadata 78dafae48a

refactor: extract magic handler f2f757def3

fix/test: ManifestEncrypted e119809951

decoder_test.go:119: expected error type [manifest
encrypted], got incorrect signature magic header:
read [17b8811b83000000] expected
[17b8811b00000000]: unexpected bytes

because the handling of the magic is extracted
into a separate function, it checked the above
error first (return value of handleMagic())

squel added 2 commits

2024-12-25 21:34:43 +00:00

fix: timestamps are unix-seconds, its not unlikely that the same second multiple manifests are created 825054bdc1

use an array of a custom struct instead e03ab6da2d

avoid having to go from keymap to array to another array

i really dont know how i like either solution tbh

makes e119809951 unnecessary

squel added 1 commit

2024-12-25 21:42:45 +00:00

apply the same multiple file input logic to decode 71249d5d9d

squel commented

2024-12-25 21:47:06 +00:00

Author

Owner

fwiw fix for e03ab6da2d 's commit message: it makes 825054bdc1 redundant, not the commit hash specified in the message

fwiw fix for e03ab6da2df48b31c92db89433c65429d4b03551 's commit message: it makes 825054bdc1dabccf274b7e2aecad9a3e0ff9fccb redundant, not the commit hash specified in the message

squel referenced this pull request

2024-12-25 21:48:15 +00:00

roadmap #8

squel changed title from ~~failure checking: handle multiple input manifests~~ to WIP: failure checking: handle multiple input manifests

2025-01-01 23:49:26 +00:00

squel added 3 commits

2025-01-07 09:36:07 +00:00

refactor: use slices.SortFunc 9d4ae43646

> "Note: in many situations, the newer slices.SortFunc function is more ergonomic and runs faster. "
https://pkg.go.dev/sort#Slice

fix: avoid race conditions; handle file not found 80b08d4a12

target file is no longer present (removed in an
update) but the old manifest still contains an
entry: warn the user

refactor: performance improvement by decoding once 91ba5012b9

squel added a new dependency

2025-01-07 09:37:50 +00:00

#15 feat: disable ANSI-colored output (NO_COLOR)

squel added 3 commits

2025-01-26 15:46:51 +00:00

rm hex.encodetostring 52e2216471

debloat mutex fa0671d2f7

cleanup b3678fd978

squel changed title from ~~WIP: failure checking: handle multiple input manifests~~ to failure checking: handle multiple input manifests

2025-01-26 15:46:55 +00:00

squel changed title from ~~failure checking: handle multiple input manifests~~ to feat: handle multiple input manifests

2025-01-26 15:47:11 +00:00

squel merged commit b3678fd978 into main

2025-01-26 15:49:41 +00:00

No reviewers

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Blocks

#15 feat: disable ANSI-colored output (NO_COLOR)

squel/steam-manifest

Reference

squel/steam-manifest!14

No description provided.

Rows
Columns