Commit graph

277 commits

Author SHA1 Message Date
76fb58e783
update models 2024-07-29 16:16:02 -07:00
baa5471af7
getting started doing hdmf mixins but am tired 2024-07-26 21:58:07 -07:00
c98f25f973
Named generic for dynamictableregions, etc. 2024-07-26 19:54:35 -07:00
3d8403e9e3
lint 2024-07-24 22:49:35 -07:00
af11bb61ec
fix imports being inside the any_of range, more manual inspection before finishing for the night 2024-07-24 22:47:06 -07:00
781b667952
working pydantic provider (at least superficially) 2024-07-24 21:51:55 -07:00
974058ce5f
assorted cleanings but i am tired 2024-07-19 21:28:24 -07:00
43187bfa3e
add numpy type to language element types 2024-07-19 20:50:48 -07:00
2e7670a2bd
yaml peek to quickly find the value of a key 2024-07-19 20:50:18 -07:00
27d18b69d8
continue stripping out old linkml monkeypatching 2024-07-18 01:01:29 -07:00
ec843a508b
update branch for linkml 2024-07-18 00:39:56 -07:00
1920c9f292
continue removing NDArray type, since it's in numpydantic now. 2024-07-10 00:20:01 -07:00
d6750f8df1
we don't need blosc tho 2024-07-10 00:09:17 -07:00
878c51e069
we do need black as a runtime dep 2024-07-10 00:05:41 -07:00
b83865352c
the hell was that 2024-07-10 00:02:39 -07:00
bf7f62fd8f
lint 2024-07-09 23:54:28 -07:00
2f59f0a69b
refactor providers into modules 2024-07-09 23:52:58 -07:00
b6af8c9718
lint 2024-07-09 03:32:37 -07:00
f4d397cde1
more docs for dataset adapters.
also:
- fix search for initial - in dataset doctest
- don't clear git repos on repeated test runs
- fix string handling in load_yaml
2024-07-09 03:26:45 -07:00
ce902476d1
working version of pretty doctests with sybil :) 2024-07-09 00:27:22 -07:00
5ccc9e6fbd
black format models 2024-07-09 00:24:08 -07:00
34c8be3a2c
functional docs build 2024-07-08 21:37:02 -07:00
fdf80666fb
regenerate models with numpydantic 2024-07-08 19:29:48 -07:00
8036b3316d
regenerate models 2024-07-08 19:12:11 -07:00
78de59a0f1
regenerate models 2024-07-08 19:10:19 -07:00
b9ed7c0026
testing custom directive for doctests 2024-07-08 19:07:21 -07:00
0cbf1ac1b2
how come ide always misses the schema lol 2024-07-03 23:36:58 -07:00
4211b6058b
improve compound dtype handling, regenerate and add 2.7.0 2024-07-03 23:36:20 -07:00
ec81032ae8
handle compound dtype 2024-07-03 20:39:49 -07:00
c9a2423a9d
missed generated schema somehow... 2024-07-03 16:00:56 -07:00
19d97ad3af
regenerate models, lint 2024-07-03 01:51:48 -07:00
01cfb54a5a
working pydantic generator 2024-07-03 01:34:24 -07:00
087064be48
adjusting array adapters to linkml arrays 2024-07-03 00:41:16 -07:00
2e40506443
requests caching, python 10 2024-07-02 00:34:01 -07:00
38c7e9ebf8
python 10 compat 2024-07-02 00:20:52 -07:00
444ecbaf04
python 10 compat 2024-07-02 00:19:21 -07:00
566d5211bb
python 10 compat 2024-07-02 00:15:18 -07:00
883c28e6d5
don't fail fast 2024-07-02 00:12:23 -07:00
1de70a1aae
python 3.10 compat & update lockfiles 2024-07-02 00:10:28 -07:00
de30f83edd
import fixtures 2024-07-02 00:01:54 -07:00
1d6584573a
one last straggler 2024-07-01 23:56:56 -07:00
f50275473c
All ruff and black fixes done 2024-07-01 23:52:32 -07:00
7c6e69c87e
ruff unsafe fixes 2024-07-01 23:05:47 -07:00
084bceaa2e
continuing manual ruff fixes 2024-07-01 22:58:55 -07:00
3768e3ce0d
enable black preview and automatic linebreaks in long strings 2024-07-01 22:12:16 -07:00
d4a0c82d00
some ruff fixes 2024-07-01 22:09:49 -07:00
07aa879bb9
ruff automatic fixes 2024-07-01 21:44:35 -07:00
79fc7f23ad
black formatting 2024-07-01 21:23:31 -07:00
3e5e73d186
add formatting config 2024-07-01 21:22:35 -07:00
a0f0b06d66
bump version 2024-07-01 19:01:15 -07:00
1eddf04400
get tests running again 2024-07-01 18:59:21 -07:00
25536c053e
docs dependencies 2024-07-01 16:35:10 -07:00
9eb4f0dec6
conversion from poetry to pdm 2024-07-01 16:19:13 -07:00
d522dd84a5
merge from origin 2024-07-01 15:46:23 -07:00
4296b27538
catchup with prior work 2024-07-01 15:44:26 -07:00
Yaroslav Halchenko
2e2ecf10c0 [DATALAD RUNCMD] Do interactive fixing of leftover ambigous typos
=== Do not change lines below ===
{
 "chain": [],
 "cmd": "codespell -w -i 3 -C 2",
 "exit": 0,
 "extra_inputs": [],
 "inputs": [],
 "outputs": [],
 "pwd": "."
}
^^^ Do not change lines above ^^^
2024-04-17 16:00:34 -04:00
Yaroslav Halchenko
63a405f4aa [DATALAD RUNCMD] run codespell throughout fixing typo automagically but ignoring the failure due to ambigous typos
=== Do not change lines below ===
{
 "chain": [],
 "cmd": "codespell -w || :",
 "exit": 0,
 "extra_inputs": [],
 "inputs": [],
 "outputs": [],
 "pwd": "."
}
^^^ Do not change lines above ^^^
2024-04-17 15:59:53 -04:00
Yaroslav Halchenko
190a5de0e3 Use _ for a variable which is not used 2024-04-17 15:55:52 -04:00
4ee97263ed
Cleanup ndarray validator
Remove outdated comments, rename inner function to match
2024-01-16 02:06:08 -08:00
eac5ef4c80 more docs work
move conftest to root so we find doctests? maybe we don't want to keep that tho
2023-10-19 21:20:50 -07:00
23a5412854 remove unused code 2023-10-19 04:01:48 -07:00
2276a7b501 docs, but need to sleep, come back in morning 2023-10-19 02:52:15 -07:00
dd956b35c3 Still working on docs!
Added ability to index datasets wtih arrays with getitem
2023-10-18 21:18:02 -07:00
af5b947d12 a bunch of docs stuff 2023-10-18 01:01:15 -07:00
464d95d64d omfg another one. i'm done 2023-10-12 00:19:40 -07:00
d2185ee1c3 for pydantic provider test by escaping additional bad chars in numpy field name 2023-10-12 00:02:52 -07:00
3101797b0d don't render jinja template as having a literal "keys" attr instead of item
also remove unnecessary warnings
2023-10-11 23:28:22 -07:00
fee86135a3 omfg i hate this bug so much if this works i'll be so mad 2023-10-11 22:38:46 -07:00
92b1a6aac0 holy hell it was a TYPE COERCION in the way linkml handles annotations and a version mismatch between CI and local
https://github.com/linkml/linkml-model/pull/162
2023-10-11 22:30:26 -07:00
0d29ed279f hold up can this really be the bug 2023-10-11 22:21:44 -07:00
338369a540 trying to debug on CI by abusing warnings lol 2023-10-11 22:07:01 -07:00
f54f5a8153 trying to debug on CI by abusing warnings lol 2023-10-11 21:57:39 -07:00
3164f92387 trying to debug on CI by abusing warnings lol 2023-10-11 21:49:45 -07:00
282acc063e trying to debug on CI by abusing warnings lol 2023-10-11 21:38:29 -07:00
3a426454c6 trying to debug on CI by abusing warnings lol 2023-10-11 21:31:00 -07:00
52754112aa make allow_repo an instance-level setting 2023-10-11 20:55:03 -07:00
38d83f3279 fix JsonObj bug https://github.com/linkml/linkml/issues/1665 2023-10-11 20:42:27 -07:00
d4493146b5 give language elements a schema annotation 2023-10-11 20:09:22 -07:00
d9d62618bc Add source namespace to all generated linkml schemas
change `namespace` annotation to `is_namespace` and use `namespace` for name of namespace
dont change commit when commit is None - otherwise always reset to recent version when we don't intend to
2023-10-11 20:03:08 -07:00
c6ca97e010 Correctly clone fixed versions in tests, adjusted fixed values to match, set default git clone temp dir from config variables 2023-10-11 19:24:08 -07:00
c4d42cdacf remove split from schema adapter because it turned out to be a bad idea, we just use tree_root instead 2023-10-11 18:32:31 -07:00
615b006e4a updating nwb_schema_language to have proper single/multi fields for author/contact 2023-10-10 01:08:25 -07:00
825b5914db allow lists and singletons in generator 2023-10-10 00:50:51 -07:00
2d1fe3ad96 read is functional again, calling it a night. 2023-10-10 00:12:33 -07:00
729da40dfd install hdf5 2023-10-09 23:40:31 -07:00
0508e3b620 re-enable truncate file test 2023-10-09 23:30:39 -07:00
91875dabb2 re-enable janky hdmf mapping test just to have it on the board for morale
Fix serializing datasets
2023-10-09 23:27:33 -07:00
f071705ac8 remove translate 2023-10-09 23:15:46 -07:00
42e64dce75 Refactor dataset adapter to use check/apply style maps!
General cleaning around maps module, getting ready to remove translate module
2023-10-09 23:06:24 -07:00
adaf939497 classes adapter 2023-10-09 20:13:42 -07:00
79397ec398 dict not list 2023-10-09 19:05:08 -07:00
2b530c4b33 don't require warning when at wrong git version 2023-10-09 19:04:00 -07:00
a7b0f8880e fix tests, cant assign to tuple 2023-10-09 18:59:53 -07:00
6f32a5bfd3 super annoying you have to import all fixtures and dependencies manually 2023-10-09 16:37:07 -07:00
edf7a4c445 [tests] - test adapter + results 2023-10-09 16:33:02 -07:00
39c600fda1 Regenerate nwb_schema_language models 2023-10-09 15:06:53 -07:00
6da6ef281f nevermind need to still fix model generation 2023-10-05 22:15:33 -07:00
06311b21bd [tests] cheap read test 2023-10-05 22:12:27 -07:00
8405ea948a [tests] ndarray proxy and serialization 2023-10-05 22:05:10 -07:00
0a9ca82476 [tests] nocover stuff that's just from the source module 2023-10-05 21:42:29 -07:00
033f6c434d [tests] nocover stuff that's just from the source module 2023-10-05 21:39:15 -07:00
c215e69fd8 [tests] nocover stuff that's just from the source module 2023-10-05 21:35:25 -07:00
a0d96014e1 [tests] pydantic generator 2023-10-05 21:22:00 -07:00
75f39b4914 huh coverage got disabled somehow 2023-10-05 18:10:04 -07:00
808c63e7c2 skip provider test just to get coverage numbers 2023-10-05 18:03:01 -07:00
d77dec5cc0 decrease deprecation warnings 2023-10-04 22:54:20 -07:00
24e0b704ff decrease deprecation warnings 2023-10-04 22:49:23 -07:00
15d1f0d7b1 clear repo imports before returning from sys.module 2023-10-04 22:32:53 -07:00
0f4f2dd683 also list linkml versions in the available versions for a pydantic provider 2023-10-04 22:27:36 -07:00
d46e86c676 return None when the module actually doesn't exist 2023-10-04 22:08:01 -07:00
5438b2489b Fix relative imports in ectopic models --
Use a MetaPathFinder to allow python to resolve imports from the temporary model directory
Build dependent versions in split builds
Clear package imports when allow_repo=False
2023-10-04 22:06:26 -07:00
3b11afded6 fix truncate hdf5 file to not try and write while iterating 2023-10-04 20:19:54 -07:00
ab63ea071c removing pandas 2023-10-04 18:02:40 -07:00
9947edfed2 Removing DataFrame type from imports, leaving as stub 2023-10-04 17:59:10 -07:00
f682105c1a it working
had to very hackily handle special cases on NWBFile class but it working. Now badly in need of a tidying, docs, and tests.
2023-10-02 23:26:43 -07:00
34f8969fa9 Like 15 validation errors away from fully reading a file.
Just going to make a special completion phase for the top-level NWBFile because it's so special cased it's not worth trying to generalize.

- Resolving hardlinked datasets in read maps
- Resolving 2nd-order empty groups
- Completing container groups
- Move fixing references to recursive function
- Multiple passes for a given phase
- NDArrayProxies should resolve dask or h5py reading on their own
- Correctly build split schema from PydanticProvider instead of needing external logic
2023-10-02 22:19:11 -07:00
eca7a5ec2e Correct schema for hdf5 path type
dont double MRO for basemodels in generated dynamictable
2023-09-28 17:58:45 -07:00
6bbf56d1a0 Make all maps exclusive
Track dependencies in build results and dont try and complete model if dependencies not alreay completed (ie. need multiple passes to complete, which is the correct behavior)
Use base models in dynamic tables, we should fix the dynamictable base class rather than just having everything be an anonymous BaseModel in order to satisfy type validation
Clear completed queue when advancing stages - we should make sure to move everything to completed with maps rather than implicitly keeping things in completed when they aren't - so we might be getting old versions of results that aren't actually fully completed but have some reference stubs still in them.
2023-09-27 20:07:39 -07:00
9fcc1458fb nearing the end - need to do final top-level packing but we're almost there! 2023-09-27 01:19:10 -07:00
a2da236b2b Regenerate models 2023-09-26 21:33:14 -07:00
8a9b6903b6 Fix generation to use actual class in type annotation even when has identifier value, while also making dicts for container classes 2023-09-26 21:30:53 -07:00
f672b931e5 Correctly handle container classes, use dicts instead of lists to be able to index by name. overriding __getitem__ and __getattr__ later 2023-09-26 20:50:11 -07:00
d16b4757e1 Use dask instead of trying to make a dataframe
compress arrays when serializing to json
2023-09-26 19:50:49 -07:00
50e816bad4 Working on finalizing the mapping operation... doing it single threaded for now and it's very slow but it completes up until the stage where we need to zip up the orphaned objects and other things that can be inferred from the model.
Need to make a proxytable model like proxyarray because reading all these tables takes way too fuckin long and it's not what we want to do anyway.
2023-09-25 22:03:53 -07:00
ce75dacf93 autogenerating models for dynamictable 2023-09-23 01:33:28 -07:00
57fa3d34a2 pandas dataframe mimic 2023-09-23 00:08:59 -07:00
aac0c7abdd need 2 stop for the night but its sort of happening 2023-09-22 02:48:40 -07:00
40984a6582 figuring out the strategy here...
- added linkml_meta classvar to store additional linkml properties if needed
- injecting path field to metaclass
- sketch of doing a queue-based read
- prune datasets & example allen institute data
2023-09-22 00:31:34 -07:00
321740f674 weird python relative import error from not having version naming 2023-09-15 01:06:50 -07:00
1f61dca46f weird python relative import error from not having version naming 2023-09-15 01:03:40 -07:00
94c204e8ea ok but really actually correct paths 2023-09-15 00:53:07 -07:00
b283e8b0bf get correct path for provider 2023-09-15 00:50:16 -07:00
ccf267babd regenerating models, some minor code cleanup and coverage omissions 2023-09-15 00:37:20 -07:00
3e2e6915cf Add ability to make JSON schema with numpy arrays! 2023-09-14 23:43:00 -07:00
e6a41415f5 successfully building many versions of nwb schema.
working on hdf5 importing, come back to it when fresh, just sorta poking at it because it's so close.
2023-09-14 02:45:01 -07:00
054ca8fd5e add monkeypatch to schemaview 2023-09-11 22:23:42 -07:00
095d0bc6a9 fixing tests by using temp directory and changing random prior commit hash from nwb core schema 2023-09-11 19:55:45 -07:00
c69cbdfb38 Docstrings and basic tests for providers.
bugfix - remove redundant language elements that cause a recursion error (eg. subtyping "float" with "float")
2023-09-11 19:38:29 -07:00
2e87fa0556 working generation from provider 2023-09-08 19:46:42 -07:00
0b0fb6c67a cute progress bar and also linkml provider works 2023-09-08 16:12:19 -07:00
43f35c50d8 fixing docs build 2023-09-07 19:39:19 -07:00
75b710c3eb docs! 2023-09-07 19:32:17 -07:00
a01bb49b1e working on providers to cache generated models 2023-09-07 18:50:50 -07:00
2b828082e0 perf enhancements to pydantic generation 2023-09-06 22:48:47 -07:00
7b97b749ef monkeypatch to fix perf problems from nptyping 2023-09-06 19:32:33 -07:00
6a9d612b41 i'm just sorta making a mess here. coming back tomorrow 2023-09-06 00:50:49 -07:00
85beef4276 trying to get absolute paths out of coverage report 2023-09-05 22:54:20 -07:00
ecbb106581 proper .coveragerc formatting 2023-09-05 22:45:26 -07:00
456a5315d2 abandoning github action build 2023-09-05 22:42:46 -07:00
ce5cd18212 move to .coveragerc 2023-09-05 22:37:41 -07:00