Skip to content

Fix storages metadata & caching #106

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 12, 2024
Merged

Fix storages metadata & caching #106

merged 5 commits into from
Apr 12, 2024

Conversation

vdusek
Copy link
Collaborator

@vdusek vdusek commented Apr 12, 2024

In the current implementation, running the following code multiple times results in errors during the parsing of the metadata files. This PR fixes it and also adds a little polishing...

import asyncio

from crawlee.request import Request
from crawlee.storages import Dataset, KeyValueStore, RequestQueue


async def main() -> None:
    dataset = await Dataset.open(name='my-dataset-42')
    await dataset.push_data({'key2': 'value'})

    kvs = await KeyValueStore.open(name='my-kvs-01')
    await kvs.set_value('key1', 'value1')

    rq = await RequestQueue.open(name='my-rq-01')
    await rq.add_request(Request.from_url('http://example.com'))


if __name__ == '__main__':
    asyncio.run(main())

@vdusek vdusek added bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team. adhoc Ad-hoc unplanned task added during the sprint. labels Apr 12, 2024
@vdusek vdusek added this to the 87th sprint - Tooling team milestone Apr 12, 2024
@vdusek vdusek requested a review from janbuchar April 12, 2024 09:31
@vdusek vdusek self-assigned this Apr 12, 2024
@github-actions github-actions bot added the tested Temporary label used only programatically for some analytics. label Apr 12, 2024
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a test that runs the code in the PR description?

@vdusek vdusek requested a review from janbuchar April 12, 2024 11:31
@vdusek vdusek merged commit a8c5d2d into master Apr 12, 2024
19 checks passed
@vdusek vdusek deleted the fix-storages branch April 12, 2024 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
adhoc Ad-hoc unplanned task added during the sprint. bug Something isn't working. t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants