34 Commits

Author SHA1 Message Date
Simon Suo
a37df8c221
Update after refactoring away parsers in LlamaIndex, also update docs to 0.6.0 API (#264) 2023-05-16 23:26:33 -04:00
Shaun M
6f745647fa
docs: update github_repo usage and example (#172) 2023-04-07 21:50:40 -07:00
akmhmgc
7d6763fa40
Modify sample code (#133) 2023-03-21 11:03:11 -07:00
ahmetkca
6072567da2
Add unittests for GHRepo reader and fix filter logic. (#104) 2023-03-13 21:09:34 -07:00
Jerry Liu
819437af8e Revert "Merge pull request #73 from ahmetkca/github-reader-test-and-fix"
This reverts commit 78bc97e9ed5e84e20d70156634b4e0ee7d612768, reversing
changes made to 13131d3e98c8be23b8a61a72098d68c5829b9a1f.
2023-03-11 17:04:01 -08:00
ahmetkca
d66823b0db Update READEME.md for github_repo 2023-03-11 14:33:27 -05:00
ahmetkca
fc255d595f Use if-block instead of match for backward compatibility. 2023-03-11 14:06:36 -05:00
ahmetkca
c7e24f1c02 Merge branch 'github-reader-test-and-fix' of https://github.com/ahmetkca/llama-hub into github-reader-test-and-fix 2023-03-09 20:46:33 -05:00
ahmetkca
381e706eca remove commented section 2023-03-09 20:45:12 -05:00
ahmetkca
37c8bb8563 cleanup 2023-03-01 03:38:39 -05:00
ahmetkca
92124b6bb5 fix import 2023-02-28 20:32:39 -05:00
ahmetkca
6e7d49dc38 fix import 2023-02-28 01:05:20 -05:00
ahmetkca
94e318af99 fix import 2023-02-28 00:53:23 -05:00
ahmetkca
e90fc59cfc correctly import from llamahub_modules 2023-02-28 00:42:24 -05:00
ahmetkca
327350d2da conditional import for llama_index and gpt_index 2023-02-28 00:30:49 -05:00
ahmetkca
1543509329 Merge remote-tracking branch 'upstream/main' into github-reader-test-and-fix 2023-02-28 00:28:39 -05:00
ahmetkca
62ea978d6c Add more test for GithubRepositoryReader and fix
Fix for filtering file extensions and directories.
Partial test coverage for GithubRepositoryReader.
Conditional import for llama_index and gpt_index
2023-02-28 00:08:01 -05:00
EmptyCrown
3f6e5af8f9 Fix test 2023-02-26 20:25:35 -08:00
EmptyCrown
c53c487921 Change readme to llama 2023-02-24 23:52:43 -08:00
EmptyCrown
79d492b3e3 Cleanup 2023-02-24 23:47:25 -08:00
EmptyCrown
19ab1afa7d First GH test
Please enter the commit message for your changes. Lines starting
:x
2023-02-24 23:44:54 -08:00
ahmetkca
5a27264db1
Add GitHub Repository Reader (#34)
* add github repository, test a new way to download loader

* test imports when downloaded from gpt_index

* Refactor(Github Repo): Move github_client and utils to modules

* Moved github_client.py and utils.py from loader_hub/github_repo to modules/github_repo
* Updated import statements in base.py to reflect the new location

* temp

* Refactor(GithubRepositoryReader): Add github_client argument

- Add github_client argument to GithubRepositoryReader constructor
- Set default value for github_client argument
- Update docstring to reflect changes

* Refactor(Github Repo): Update init file

- Remove imports of base, github_client and utils
- Add imports of GithubRepositoryReader and GithubClient
- Update __all__ to include the new imports

* Fix(library): Update library.json

- Updated library.json to include __init__.py file

* Refactor(GithubRepositoryReader): Add filter for directories and files

- Add filter for directories and files in GithubRepositoryReader
- Ignore directories and files that do not pass the filter
- Print out if directory or file is ignored due to filter

* Refactor(BaseReader): Check filter files

- Refactor `_check_filter_files` to `_check_filter_file_extensions` in `BaseReader`
- Ignoring files due to filter

* Docs(FilterType): Add documentation for FilterType enum

- Add documentation for FilterType enum
- Explain what the enum is used for
- Describe the attributes of the enum

* Add(GPT Index): Add GPT Index example

Add GPT Index example to README
- Set OPENAI_API_KEY environment variable
- Download GithubRepositoryReader module
- Create GithubClient and GithubRepositoryReader
- Load data from Github Repository
- Create GPTSimpleVectorIndex
- Query the index

* Add(GPT Index): Add GPT Index example

Add GPT Index example to README
- Set OPENAI_API_KEY environment variable
- Download GithubRepositoryReader module
- Create GithubClient and GithubRepositoryReader
- Load data from Github Repository
- Create GPTSimpleVectorIndex
- Query the index

* Add(GPT Index): Add GPT Index example

Add GPT Index example to README
- Set OPENAI_API_KEY environment variable
- Download GithubRepositoryReader module
- Create GithubClient and GithubRepositoryReader
- Load data from Github Repository
- Create GPTSimpleVectorIndex
- Query the index

* change the import path for extras

* change import path for extra files to absolute

* Add test for GithubClient currently not using mocks which is not ideal

* Update test_github_reader.py

* Update test_github_reader.py

---------

Co-authored-by: Jesse Zhang <jessetanzhang@gmail.com>
2023-02-24 23:41:48 -08:00
ahmetkca
2adbb24a2c change import path for extra files to absolute 2023-02-14 23:42:18 -05:00
ahmetkca
e12e7f5cd9 change the import path for extras 2023-02-14 23:31:05 -05:00
ahmetkca
cdccaec8b8 Add(GPT Index): Add GPT Index example
Add GPT Index example to README
- Set OPENAI_API_KEY environment variable
- Download GithubRepositoryReader module
- Create GithubClient and GithubRepositoryReader
- Load data from Github Repository
- Create GPTSimpleVectorIndex
- Query the index
2023-02-13 20:44:09 -05:00
ahmetkca
14cfdf75b1 Docs(FilterType): Add documentation for FilterType enum
- Add documentation for FilterType enum
- Explain what the enum is used for
- Describe the attributes of the enum
2023-02-13 03:49:43 -05:00
ahmetkca
de761cd341 Refactor(BaseReader): Check filter files
- Refactor `_check_filter_files` to `_check_filter_file_extensions` in `BaseReader`
- Ignoring files due to filter
2023-02-13 01:49:31 -05:00
ahmetkca
91c6bf33cd Refactor(GithubRepositoryReader): Add filter for directories and files
- Add filter for directories and files in GithubRepositoryReader
- Ignore directories and files that do not pass the filter
- Print out if directory or file is ignored due to filter
2023-02-13 01:46:58 -05:00
ahmetkca
093fc689d7 Refactor(Github Repo): Update init file
- Remove imports of base, github_client and utils
- Add imports of GithubRepositoryReader and GithubClient
- Update __all__ to include the new imports
2023-02-13 01:18:12 -05:00
ahmetkca
1bf025dd31 Refactor(GithubRepositoryReader): Add github_client argument
- Add github_client argument to GithubRepositoryReader constructor
- Set default value for github_client argument
- Update docstring to reflect changes
2023-02-13 01:02:47 -05:00
ahmetkca
0f82eb0e57 temp 2023-02-12 04:01:39 -05:00
ahmetkca
6227d7496e Refactor(Github Repo): Move github_client and utils to modules
* Moved github_client.py and utils.py from loader_hub/github_repo to modules/github_repo
* Updated import statements in base.py to reflect the new location
2023-02-12 03:52:27 -05:00
ahmetkca
f36f2e795e test imports when downloaded from gpt_index 2023-02-12 03:28:32 -05:00
ahmetkca
fbffd7e4fa add github repository, test a new way to download loader 2023-02-12 03:04:12 -05:00