Commit graph

75 commits

Author SHA1 Message Date
Robin Ward
b42f28d4c3 Improved mailing list import.
Now uses a SQLite database to store messages rather than JSON files
for performance and memory considerations.
2016-06-14 11:45:17 -04:00
Neil Lalonde
f07b3e7df9 PERF: base importer updates user_stats with sql (mostly) 2016-05-17 18:39:04 -04:00
Régis Hanol
b716886240 update bbPress importer 2016-03-07 18:22:15 +01:00
Gerhard Schlager
8a486d8cea Allow importers to set empty names 2016-02-21 23:35:47 +01:00
Gerhard Schlager
eb166e78b6 Don't try to import invalid websites 2016-02-21 17:38:04 +01:00
Neil Lalonde
08e10c2a9d FEATURE: Zoho importer 2015-12-04 15:06:46 -05:00
Régis Hanol
26006f12a2 Increased 'error resistance' in base importer class 2015-12-01 15:38:21 +01:00
Robin Ward
6c6d406e63 Small fixes to import base. 2015-10-29 16:47:27 -04:00
Gerhard Schlager
6238a43f93 Fix base and vBulletin importer 2015-10-21 19:07:31 +02:00
Sam
606aeb9d55 improvements to importer
- improve perf of test for existing posts
- always use a system guardian when importing posts
- for lithuim importer requery raw (transform is not repeatable)
2015-10-16 10:41:21 +11:00
Gerhard Schlager
29d27ec2ef FIX: Prevent cast error when import_id contains strings and numbers
Some importers insert numbers and strings as import_id into the same
custom_fields table. This change prevents cast errors during imports.
2015-09-30 16:38:06 +02:00
Kane York
821124f3fd FEATURE: Skip batches if all records exist
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
Kane York
200ee15991 FIX: Importer not handling usernames correctly 2015-09-21 16:27:47 -07:00
Gerhard Schlager
8c03dd16af Adds support for importing password hashes used by "migratepassword" plugin
Adds setting to phpBB3 importer for importing passwords (default: off)
Plugin: https://github.com/discoursehosting/discourse-migratepassword
2015-08-20 22:22:47 +02:00
Robin Ward
bb93a345eb UX: Use smaller messages for moderator actions. 2015-07-27 10:09:21 -04:00
Gerhard Schlager
2dd01c61b0 Improves the base importer
- Move some methods into their own classes in order to make it easier
  to reuse them outside of classes extending the base importer. For
  compatibility reasons the old methods are still in the base importer
  and delegate to the new objects. The following methods and hashes were
  extracted:
    - all the lookup maps for existing and imported data
    - all the methods used for uploads and attachments
- No need to store failed users and groups. This information wasn't
  used anyway.
- Print progress instead of category names when importing categories.
- Allow importers to override if bbcode_to_md should be used (until now
  it always used ARGV)
- Allow importers to add additional site settings that automatically get
  restored after the importer finishes.
- Show how many posts and messages are imported per minute. This should
  help detecting when the import is slowing down and needs to be
  restarted.
- Use max_image_width and max_image_height from settings instead of
  hard-coded values for uploaded images.
2015-07-16 15:28:42 +02:00
Sam
bfd28ca9a9 ignore deleted ... too much trouble 2015-07-16 15:11:28 +10:00
Régis Hanol
dcc4267490 FIX: base script import to handle category color/text color 2015-05-19 12:40:35 +02:00
Neil Lalonde
e8527c5e3e it's ok if import script can't lower trust level of an admin user 2015-04-17 11:34:21 -04:00
Régis Hanol
78f5fe190a FIX: base importer should warn when there's an error in the user_profile 2015-04-17 16:31:46 +02:00
Neil Lalonde
f8080c0138 add a spinner for importers 2015-03-30 12:29:48 -04:00
Neil Lalonde
b1d78a4fd5 FIX: use batches for speed, and show progress from importers' update_tl0 2015-03-20 17:05:14 -04:00
Régis Hanol
5084e2bdf1 Merge pull request #3284 from techAPJ/patch-1
FEATURE: new site setting min_first_post_length
2015-03-19 19:55:27 +01:00
Arpit Jalan
b706307ac7 FEATURE: new site setting min_first_post_length 2015-03-20 00:20:38 +05:30
Gerhard Schlager
9913dfa6c6 Imported users shouldn't get an email immediately after the import
This sets the last_emailed_at for imported users to now which postpones the sending of digest emails for the configured amount of time. Therefore, the first digest email gets sent after 1 week (the default value of default_digest_email_frequency).
As suggested in https://meta.discourse.org/t/dont-trigger-digests-on-migration/26345/7
2015-03-18 21:02:40 +01:00
Gerhard Schlager
53591ade00 Adds a few enhancements to the base importer
- Allows importing globally pinned topics
- Doesn't restore the original value of a SiteSetting anymore if an importer changed its value
- Allows all file extensions during the import
- Adds the ability to import bookmarks
2015-03-13 21:24:11 +01:00
Gerhard Schlager
b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
Michael C. Grant
6030fe1f9d Added location.present? test to user profile updating. 2015-03-07 08:42:42 -06:00
Neil Lalonde
de167165e3 FIX: don't change default_digest_email_frequency during import 2015-02-20 10:30:30 -05:00
Neil Lalonde
ae2f861101 FEATURE: import scripts update last_seen_at on users based on created date and last post date 2015-02-12 12:25:01 -05:00
Neil Lalonde
07af79c85b Ning import: users who have never posted are trust level 0 2015-02-06 16:03:41 -05:00
Régis Hanol
0e5c9b2590 small upload code refactor 2015-02-03 18:44:18 +01:00
Gerhard Schlager
827daf7f0f FIX: The order of includes in the base importer was wrong.
https://meta.discourse.org/t/importer-for-simple-machines-2-forums/17656/58
2015-01-31 15:42:39 +01:00
Gerhard Schlager
72755970c6 Improve the base importer
- FEATURE: Print the time it took for the import to finish
- FEATURE: Allow importing of unactivated users
- FIX: Remove unused variables
- FIX: Accessing an undefined variable is not a good idea
2015-01-23 22:44:00 +01:00
Neil Lalonde
a2099110aa FEATURE: Ning importer 2015-01-23 12:19:46 -05:00
Régis Hanol
6c4d852011 Improve vBulletin importer
- FEATURE: TopicCreator now supports 'pinned_at' parameter
- FIX: 🐛 FIX TopicQuerySQL to support pinned topic older than 2010
- FIX: 🐛 Properly remove all HTML Entities from Usernames/Titles/Category Names/Groups in vBulletin importer
- FIX: 🐛 Properly handle specific vBulletin BBCode (quotes/mentions)
- FIX: 🐛 Make sure we generate a username from the name of the user instead of a fake email
- FEATURE: Allow for custom timezone in vBulletin importer
- FEATURE: Support for profile pictures/background in vBulletin importer
- FIX: 🐛 merge the categories tree to only 2 levels in vBulletin importer
2015-01-19 15:00:55 +01:00
Sam
1d4daca75d add override for post creation in importer 2014-10-31 15:19:40 +11:00
Neil Lalonde
cea2fe5ee9 FEATURE: import attachments in phpbb3 importer 2014-10-07 16:47:41 -04:00
Robin Ward
47573fcd99 FIX: error importing some topics where bumped_at coudln't be
determined.
2014-10-06 09:26:52 -04:00
Robin Ward
f4f54fd49b First stab at Importer for Drupal QA 2014-09-10 14:27:40 -04:00
Régis Hanol
78bcea5247 PERF: improve perf on consistency query in importers 2014-09-08 22:08:41 +02:00
Régis Hanol
9964955e88 FEATURE: fix consistency of users.last_posted_at in importers 2014-09-08 19:36:55 +02:00
Sam
59d04c0695 Internal renaming of elder,leader,regular,basic to numbers
Changed internals so trust levels are referred to with

TrustLevel[1], TrustLevel[2] etc.

This gives us much better flexibility naming trust levels, these names
are meant to be controlled by various communities.
2014-09-05 15:20:52 +10:00
Régis Hanol
7a7a38c78e FIX: always show the progress bars when skipping posts/users in importers 2014-09-04 19:18:22 +02:00
Régis Hanol
a0569a22be FEATURE: reset topic counters at the end of the import 2014-09-04 19:08:57 +02:00
Régis Hanol
85cbb001ae FIX: properly close topics in vBulletin importer
FEATURE: add backtrace when an exception happen (importers)
FIX: post-processing should also happen on first posts (vBulletin
importer)
PERF: faster topic bypass when already imported
2014-09-04 17:55:05 +02:00
Gerhard Schlager
5a769c7c48 FIX: Reset changed site settings after import is complete
After an import has finished the site settings should be reset to their
previous values. For example, since 'disable_emails' was set to false
before any import, it wasn't possible to login after the import
(sending email for password reset didn't work).
2014-08-31 18:09:21 +02:00
Régis Hanol
9b86d1507d VBULLETIN: add some stats about the imported groups 2014-08-25 12:33:08 +02:00
Régis Hanol
7aaf718cf3 update bbcode dialect and fix vBulletin importer 2014-08-25 10:48:29 +02:00
Régis Hanol
76824cda31 FIX: completely disable emails in importers 2014-08-23 11:12:01 +02:00