Skip to content

Export Wordpress #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 24 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
41ced26
Ingesting updates from Wordpress:
liquidsteves Aug 31, 2020
e356d94
Ingesting updates from Wordpress:
liquidsteves Aug 31, 2020
7eb1bf0
Update shared repository
liquidsteves Sep 11, 2020
8e4a7ed
update requirements.txt
liquidsteves Sep 20, 2020
f304b24
Merge pull request #16 from nyc-cto/master
liquidsteves Sep 21, 2020
499a268
Merge branch 'steve/ingest-wordpress' of https://github.com/nyc-cto/t…
liquidsteves Sep 21, 2020
7106adb
Test test test
liquidsteves Sep 21, 2020
cbabeeb
Test test test
liquidsteves Sep 21, 2020
a2311cc
use shannon utils for git_push
liquidsteves Sep 21, 2020
43e2c96
connect to serge
liquidsteves Sep 21, 2020
7ce6e87
Update shared repository: wordpress
liquidsteves Sep 22, 2020
da75d8a
Update shared repository: wordpress
liquidsteves Sep 22, 2020
9b892c2
Update shared repository: wordpress
liquidsteves Sep 22, 2020
ad1abe6
Fix elements
nyccto-rapicastillo Sep 22, 2020
d9c8ec3
Wordpress
nyccto-rapicastillo Sep 22, 2020
10d58e5
Wordpress ingest
nyccto-rapicastillo Sep 25, 2020
7982330
Add Wordpress Export
nyccto-rapicastillo Sep 25, 2020
795b8a7
Merge branch 'master' into rapi/export-wordpress
nyccto-rapicastillo Sep 25, 2020
f7a0883
Export doc for wordpress
nyccto-rapicastillo Sep 25, 2020
301c851
Export doc for wordpress
nyccto-rapicastillo Sep 25, 2020
a7ebc68
Update translation
nyccto-rapicastillo Sep 25, 2020
f1b2dc8
Merge pull request #26 from nyc-cto/rapi/update-wp-translations
nyccto-rapicastillo Sep 25, 2020
71db715
Update translation
nyccto-rapicastillo Sep 25, 2020
82d8c9f
Merge branch 'rapi/export-wordpress' of github.com:nyc-cto/tms into r…
nyccto-rapicastillo Sep 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 56 additions & 1 deletion ingestion/ingest-wordpress.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import base64
import json
import os.path
import os
import requests

import sys
Expand Down Expand Up @@ -42,7 +43,26 @@ def get_posts():
def do_work():
for post in get_posts():
write_post(post)
utils.git_push(utils.PROJECT_ROOT_GIT_PATH, commit_message="Update shared repository: wordpress", enable_push=True)
utils.git_push(utils.PROJECT_ROOT_GIT_PATH, commit_message="Update shared repository: wordpress", enable_push=False)


def do_export(id, lang, message):
"""
This sends the tranlated item
"""
export_url = os.environ.get('EXPORT_URL')

url = f"{export_url}/wp-json/elsa/v1/translate/{id}/{lang}"

user = os.environ.get('EXPORT_USER')
password = os.environ.get('EXPORT_PASSWORD')
credentials = user + ':' + password
token = base64.b64encode(credentials.encode())
header = {'Authorization': 'Basic ' + token.decode('utf-8')}

response = requests.post(url, headers=header, json=message)
return {"status": "success"}


app = Flask(__name__)
api = Api(app)
Expand All @@ -56,7 +76,42 @@ def post(self):
do_work()
return "Post"

class WordpressExportListener(Resource):
"""
`WordpressExportListener` will be the endpoint to be triggered when we want to return something back to a
wordpress site. It requires the following body attributes:


JSON
{
"id": ID of the source content,
"lang": target language code (ex. "fr"),
"content": HTML content (ex. "<p>Bonjour le monde! Je suis Rapi Castillo</p>"),
"title": Title of the translated content (ex. "Bonjour le monde!"),
"excerpt": Excerpt of the translated content (ex. "Hello this is a french translation xx"),
"status": "publish"
}

This will create/update the translated content.

"""
def post(self):

post_data = request.json
id=post_data.get("id")
lang=post_data.get("lang")
message = {
"content": post_data.get("content"),
"title": post_data.get("title"),
"excerpt": post_data.get("excerpt"),
"status": post_data.get("status")
}

return do_export(id, lang, message)
Comment on lines +98 to +110
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than receiving the text of the message from a post() parameter, we need to get it from the file system. For instance, here are some examples of spanish (es) output files.
https://github.com/nyc-cto/tms-data/tree/aditya/development/local/source_files/es
I've been using the "wp" prefix to indicate wordpress files. Can you read the json from them?

In our Docker setup, this tms-data git branch gets checked out under /var/tms-data/source_files.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re: actually triggering the code to run -- Ideally we need our Serge wrapper to hit your rest api here. Can you ask Aditya about the best place for the call to happen? If there's no good place to put the hook, we can instead use a celery/cron-ish timer to run your code periodically

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds good. I'll check in with Aditya.


api.add_resource(WordpressUpdateListener, "/wp-updates")
api.add_resource(WordpressExportListener, "/wp-export")


if __name__ == "__main__":
# Warning: When debug mode is unsafe. Attackers can use it to run arbitrary python code.
Expand Down
2 changes: 1 addition & 1 deletion serge/configs/base_config.serge
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ jobs
# plugin for more information.
data
{
path_matches \/(about|title|description)$
path_matches \/(about|wptitle|content|description)$
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you remove this? You inherited this from my branch, but I took it out.

#param1 value1
#param2 value2
}
Expand Down
1 change: 1 addition & 0 deletions shared_directory/en/wp1.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"id": 1, "wptitle": {"rendered": "Hello world!"}, "content": {"rendered": "\n<p>Welcome to WordPress. &#8230; This is your first post. Edit or delete it, then start writing!</p>\n", "protected": false}, "link": "http://localhost:8888/2020/04/20/hello-world/", "modified_gmt": "2020-09-22T16:01:56"}
1 change: 1 addition & 0 deletions shared_directory/en/wp31.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"id": 31, "wptitle": {"rendered": "Goodbye"}, "content": {"rendered": "\n<p>Welcome to WordPress. &#8230; This is your second post. Edit or delete it, then start writing?</p>\n", "protected": false}, "link": "http://localhost:8888/2020/09/09/test-new-post/", "modified_gmt": "2020-09-22T15:58:53"}
118 changes: 118 additions & 0 deletions wordpress/wp-elsa/wp-elsa.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
<?php
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So are Cyber Command / DoITT ok with us installing this custom code on our wpengine instance? What is the approval process? Do we need to go through an approval process every time we change the code?

If I wanted to add my own custom php function (say, manually building my own simple "webhook" by hitting a hard coded endpoint every time a wordpress post is edited), would that be ok?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, according to @jisaf the process of having this go through Cyber would be more streamlined compared to if we are to use plugins that are not within WPEngine because it's worked by an unapproved third party


/***
*
* CREATE TRANSLATION
* Receiving new content via POST
*/

add_action('rest_api_init', 'elsa_translation_endpoints');
function elsa_translation_endpoints() {
register_rest_route(
'elsa/v1', '/translate/(?P<id>\d+)/(?P<lang>\w+)',
array(
'methods' => 'POST',
'callback' => 'create_translation',
'args' => array(
'id' => array(
'validate_callback' => function($param, $request, $key) {
return is_numeric( $param );
}
),
'lang'
),
)
);
}


/**
* create_translation
*
* This processes the request from the /translate endpoint and creates
* a translated content.
*/
function create_translation(WP_REST_Request $request) {

$language = $request['lang'];
$id = $request['id'];

$current_language = pll_get_post_language($id);

$translated_post = pll_get_post($id, $language);
if ($translated_post == null) {
// Create Translation
$json = json_decode($request->get_body());
$json->id = $id;
$json->lang = $language;

$new_content = wp_insert_post(
array(
'post_title' => $json->title,
'post_excerpt' => $json->excerpt,
'post_content' => $json->content
)
);


pll_set_post_language($new_content, $json->lang);

pll_save_post_translations(array($current_language => $id, $language => $new_content));

$content = new WP_Query(array('p' => $new_content, 'post_type' => 'any'));
$response = new WP_REST_Response(
$content
);

$response->set_status(201);
return $response;
} else {
// Update the translation
$json = json_decode($request->get_body());
$json->id = $translated_post;

$time = strtotime( 'now' );
$content = array(
'ID' => $json->id,
'post_title' => $json->title,
'post_excerpt' => $json->excerpt,
'post_content' => $json->content,
'post_date' => date( 'Y-m-d H:i:s', $time ),
'post_date_gmt' => gmdate( 'Y-m-d H:i:s', $time ),
);

$updated_post = wp_update_post($content);

$response = new WP_REST_Response(
$content
);

$response->set_status(201);
return $response;
}
}




/**
* elsa_requireents_activate
*
* This function would require wordpress to have Polylang installed when acivating the plugin.
*/
function elsa_requirements_activate() {

if ( current_user_can( 'activate_plugins' )
&& !(
function_exists( 'pll_get_post_language')
&& function_exists('pll_save_post_translations')
)
) {
// Deactivate the plugin.
deactivate_plugins( plugin_basename( __FILE__ ) );
// Throw an error in the WordPress admin console.
$error_message = '<p style="font-family:-apple-system,BlinkMacSystemFont,\'Segoe UI\',Roboto,Oxygen-Sans,Ubuntu,Cantarell,\'Helvetica Neue\',sans-serif;font-size: 13px;line-height: 1.5;color:#444;">' . esc_html__( 'This plugin requires ', 'simplewlv' ) . '<a href="' . esc_url( 'https://wordpress.org/plugins/simplewlv/' ) . '">Polylang</a>' . esc_html__( ' plugin to be active.', 'simplewlv' ) . '</p>';
die( $error_message ); // WPCS: XSS ok.
}
}
register_activation_hook( __FILE__, 'elsa_requirements_activate' );